Test Automation and EasyRepro: 05 – Adding EasyRepro Tests to Azure DevOps

by Contributed | Apr 26, 2021 | Technology

This article is contributed. See the original author and article here.

The following is the fifth and final on a series of articles by @Ali Youssefi that we have been cross-posting into this Test Community Blog for a couple of months now. These articles were first published by Ali in the Dynamics community but since the topic related with Testing, Quality and Selenium, we though it would makes sense to publish here as well.

If you didn’t get a chance to catch the previous parts of these series, please have a look the links below: 

• Test Automation and EasyRepro: 01 – Overview and Getting Started


• Test Automation and EasyRepro: 02 – Designing and Debugging Unit Tests


• Test Automation and EasyRepro: 03 – Extending the EasyRepro Framework


• Test Automation and EasyRepro: 04 – Monitoring and Insights with EasyRepro

Otherwise, please read ahead!

Summary

EasyRepro is an open source framework built upon Selenium allowing automated UI tests to be performed on a specific Dynamics 365 organization. This article will cover incorporating EasyRepro into a DevOps Build pipeline, allowing us to begin the journey toward automated testing and quality. We will cover the necessary settings for using the VSTest task and configuring the pipeline for continuous integration and dynamic variables. Finally we will review the test result artifacts providing detailed information about each unit test and test run.

Getting Started

If you haven’t already, please review the previous articles showing how to create, debug and extend EasyRepro. This article assumes that the steps detailed in the first article titled Cloning locally from Azure DevOps have been followed. This approach will allow our test design team to craft and share tests for quality assurance across our DevOps process.

The Run Settings File

The run settings file for Visual Studio unit tests allow variables to be passed in similar to the app.config file. However this file is specific to Visual Studio tests and can be used in the command line and through Azure DevOps pipelines and test plans. Here is a sample to create a runsettings file.

The image above shows how to implement the TestRunParameters needed for EasyRepro unit tests. You can also find an example in the Microsoft.Dynamics365.UIAutomation.Sample project called easyrepro.runsettings. The runsettings file can be used to set the framework version, set paths for adapters, where the result artifacts will be stored, etc. In the Settings section below we will point to a runsettings file for use within our pipeline.

The ClassInitialize Data Attribute

The ClassInitialize data attribute is used by Visual Studio unit tests to invoke the constructor of a test class. This decoration coupled with the runsettings file will allow us to pass in a TestContext object containing the run parameters.

Properties

The configuration values from the runsettings file are included in the Properties object similar to the app.config file. For usage with EasyRepro we will want to leverage the .ToSecureString extension method which will help us when logging into the platform. Below is an example using this extension method.

Setting up the Build Pipeline

In the first article Test Automation and EasyRepro: 01 – Getting Started, we discussed how to clone from GitHub to a DevOps Git repository which we can then clone locally. The examples below follow this approach and assume you have cloned locally from Azure DevOps Git repo.

The first thing to setting up the Build Pipeline is to navigate to the DevOps organization and project. Once inside the project, click on the Pipelines button to create a Build Pipeline. Tasks here require resolving the NuGet package dependencies, building a solution with MS Build and running unit tests using the VS Test task as shown in the image below.

The core task is the VsTest which will run our Visual Studio unit tests and allow us to dynamically pass values from the Build pipeline variables or from files in source control. The section below goes into the VsTest task, specifically version 2.0.

Reviewing the VsTest task

Test Files

The test file field needs to point to a working directory, dictated by the Search folder field, to locate the compiled unit test assemblies. When using the default task this field looks for assemblies with the word test. If you’re starting with the Sample project from EasyRepro you will need to change this to look for the word Sample as shown above. When this task is run you can confirm if the correct assembly is found in the log for the task.

Test Filter Criteria

The test filter criteria field is used to help limit the scope of the unit tests run within the unit test assembly. Depending on the data attribute decorations you can restrict the unit test run task to only run specific tests. The criteria can be somewhat challenging if you haven’t worked with them before so I’d suggest using the Visual Studio Command Prompt to test locally to better understand how this will work in Azure DevOps Pipelines.

The above image shows an example of using the TestCaseFilter argument to run a specific test class. This argument can be used to run specific classes, priorities, test categories, etc. For instance

More information on the test filter criteria can be found here.

Settings File

The settings file field works with the vstest.console.exe utilizing the “/Settings” argument but allows the ability to pick a file from the repository directly. This field is customizable to also work with Build Pipeline variables which I’ll describe next.

Override Test Run Parameters

Overriding test run parameters is useful when we want to reuse the same test settings but pass in variables from the Build Pipeline. In the example below I’m replacing the parameters from the runsettings file on the left with Build Pipeline variables on the right.

Below are my pipeline variables I’ve defined. This allows me to check in a runsettings file but not have to modify parameters before committing. The values can be plain or secure strings which you will have to take into account if you plan to use one or the other. These variables can also be provided at run time when we queue the build.

Enabling the Continuous Integration Trigger

Enabling the continuous integration trigger allows developers to craft their unit tests and have our build pipeline run on push of a commit. This is configured from the Triggers tab on the pipeline which will bring up the above image. To enable, check the box for ‘Enable continuous integration’ and set which branch you want to have this fire off on. This doesn’t restrict a tester from queuing a build on demand but does help us move forward towards automation!

Running the Build Pipeline

To kick off the build pipeline, commit and push changes to your unit tests as you would any Git commit. Once the push is finished you can navigate to the Azure DevOps org and watch the pipeline in action. Below is a recording of a sample run.

Exploring the Results File

The results of the unit tests can be found in the build pipeline build along with the logs and artifacts. The test results and artifacts are also found in the Test Runs section of the Azure Tests area. The retention of these results are configurable within the Azure DevOps project settings. Each build can be reviewed at a high level for various test result statuses as shown below:

The summary screen includes the unit tests ran and information about failed tests that can be used to track when a regression began. Columns shown include the last time a test ran and what build it began to fail on.

When investigating a failed unit test a link to a new or existing bug can be added. This is useful to help track regressions and assign to the appropriate team. Bugs can be associated from the test run or specific unit test and include links to the build, test run and test plan. The exception message and stack trace are automatically added if linked from the failed unit test.

Each test run includes a test results file that can be downloaded and viewed with Visual Studio. The test artifacts can also be retained locally for archiving or reporting purposes. The contents of the test can be extracted and transformed to be used by platforms such as PowerBI or Azure Monitor.

Its key to point out that when used with a Azure Test Run these results can be retrieved via the API and reported on directly. Below is an image of the response from the Test Run.

Next Steps

Including Unit Tests as a Continuous Delivery Quality Gate

Building and running our EasyRepro tests with Build Pipelines represent an important first step into your journey into DevOps as well as what has been called Continuous Quality. Key benefits here include immediate feedback and deep insights into high value business transactions. Including these types of tests as part of the release of a Dynamics solution into an environment is paramount to understanding impact and providing insight.

Including Code Analysis as a Continuous Delivery Quality Gate

One thing I’d like to point out is that UI testing can help ensure quality but this should be coupled with other types of testing such as security testing or code analysis. The PowerApps Product Group has a tremendously valuable tool called the PowerApps Project Checker for code analysis. Project Checker can help identify well documented concerns that may come up as part of our deployment. This tool can be used via PowerShell or from the PowerApps Build Tasks within the Visual Studio Marketplace. It can also be run from the PowerApps portal in a manual fashion if desired.

• Make higher quality Apps with Solution checker


• Automatically validate your solutions using the PowerApps checker PowerShell Module


• Best practice rules used by solution checker
  

   

  
  

   

  
  

This code quality step can be included as part of the extraction of modifications from your development or sandbox environments or as a pre step before packaging a managed solution for deployment. For additional detail there is a wonderful post by the Premier Field Engineer team documenting how to include this task into your pipelines. Special thanks to Premier Field Engineers’ Paul Breuler and Tyler Hogsett for documenting and detailing the journey into DevOps.

I highly recommend incorporating this important step into any of your solution migration strategies even if you are still manually deploying solutions to better understand and track potential issues.

Scheduling Automated Tests

Scheduling tests can be done in various ways, from build and release pipelines to Azure Test Plans. For pipelines triggers can be used to schedule based off a predetermined schedule. Azure Test Plans allow for more flexibility to run a specific set of tests based off of test cases linked to unit tests. To find out more about setting this up, refer to this article.

Conclusion

This article covers working from a local version of EasyRepro tests and how to incorporate into a Azure DevOps Build Pipeline. It demonstrates how to configure the VsTest task, how to setup triggers for the Build Pipeline and how to review Test Results. This should be a good starting point into your journey into DevOps and look forward to hearing about it and any question you have in the comments below.

At the Microsoft Test Team, we have followed this tutorial and often use EasyRepro for our UI-Test related needs for Dynamics 365, please stay tune for a Tips and Trick related to EasyRepro, until next time!

Best practices for leveraging Microsoft 365 Defender API's – Episode Three

by Contributed | Apr 26, 2021 | Technology

This article is contributed. See the original author and article here.

In the previous episode, we described how you can easily use PowerBi to represent Microsoft 365 data in a visual format. In this episode, we will explore another way you can interact with the Microsoft 365 Defender API. We will describe how to automate data analysis and hunting using Jupyter notebook.

Automate your hunting queries 

While hunting and conducting investigations on a specific threat or IOC, you may want to use multiple queries to obtain wider optics on the possible threats or IOCs in your network. You may also want to leverage queries that are used by other hunters and use it as a pivot point to perform deep analysis and find anomalous behaviors. You can find a wide variety of examples in our Git repository where various queries related to the same campaign or attack technique are shared.  

In scenarios such as this, it is sensible to leverage the power of automation to run the queries rather than running individual queries one-by-one.  

This is where Jupyter Notebook is particularly useful. It takes in a JSON file with hunting queries as input and executes all the queries in sequence. The results are saved in a .csv file that you can analyze and share. 

Before you begin 

JUPYTER NOTEBOOK 

If you’re not familiar with Jupyter Notebooks, you can start by visiting https://jupyter.org for more information. You can also get an excellent overview on how to use Microsoft 365 APIs with Jupyter Notebook by reading Automating Security Operations Using Windows Defender ATP APIs with Python and Jupyter Notebooks.   

VISUAL STUDIO CODE EXTENSION 

If you currently use Visual Studio Code, make sure to check out the Jupyter extension.  

Figure 1. Visual Studio Code – Jupyter Notebook extension 

Another option to use Jupyter Notebook is the Microsoft Azure Machine Learning service. 

Microsoft Azure Machine Learning is the best way to share your experiment with others and for collaboration. 

Please refer to Azure Machine Learning – ML as a Service | Microsoft Azure for additional details. 

Figure 2. Microsoft Azure Machine Learning 

In order to create an instance, create a resource group and add the Machine Learning resource. The resource group lets you control all of the resources from a single entry point. 

Figure 3. Microsoft Azure Machine Learning – Resource 

When you’re done, you can run the same Jupyter Notebook you are running locally on your device.  

Figure 4. Microsoft Azure Machine Learning Studio 

App Registration 

The easy way to access the API programmatically is to register an app in your tenant and assign the required permissions. This way, you can authenticate using the application ID and application secret. 

Follow these steps to build your custom application. 

• Connect to https://portal.azure.com/ 


• App registration 

Figure 5. App registration 

Select “NEW REGISTRATION“. 

Figure 6. Register an application 

Provide the Name of your app, for example, MicrosoftMTP, and select Register. 

Once done, select “API Permission“. 

Figure 7. API Permissions 

Select “Add a permission“. 

Figure 8. Add permission 

Select the “APIs my organization uses“. 

  Figure 9. Alert Status 

Figure 10. Request API permission 

Search for Microsoft Threat Protection and select it. 

Figure 11. Microsoft Threat Protection API 

Select “Application Permission“. 

Figure 12. Application Permissions 

Then select: 

• AdvancedHunting.Read.All 


• Incident.Read.All 

Figure 13. Microsoft 365 Defender API – Read permission 

Once done select “Add permissions“. 

Figure 14. Microsoft 365 Defender API – Add permission 

Get Started 

Now that we have the application ready to access the API via code, let’s try to see is any of the Qakbot queries shared in Microsoft 365 Defender Git produce any results. 

Figure 15. Microsoft 365 Defender – Hunting Queries 

The following queries will be used in this tutorial:  

Javascript use by Qakbot malware 

Process injection by Qakbot malware 

Registry edits by campaigns using Qakbot malware 

Self-deletion by Qakbot malware 

Outlook email access by campaigns using Qakbot malware 

Browser cookie theft by campaigns using Qakbot malware 

Detect .jse file creation events 

We need to grab the queries that we want to submit and populate a JSON file with this format. Please be sure that you are properly managing the escape character in the JSON file (if you use Visual Studio Code (VSCode) you can find extensions that can make the ESCAPE/UNESCAPE process easiest, just pick your favorite one). 

[ 
        { 
            "Description": "Find Qakbot overwriting its original binary with calc.exe", 
            "Name": "Replacing Qakbot binary with calc.exe", 
            "Query": "DeviceProcessEvents | where FileName =~ "ping.exe" | where InitiatingProcessFileName =~ "cmd.exe" | where InitiatingProcessCommandLine has "calc.exe" and InitiatingProcessCommandLine has "-n 6" and InitiatingProcessCommandLine has "127.0.0.1" | project ProcessCommandLine, InitiatingProcessCommandLine, InitiatingProcessParentFileName, DeviceId, Timestamp", 
            "Mitre": "T1107 File Deletion", 
            "Source": "MDE" 
        } 
] 

Once you have all your queries properly filled, we must provide the following parameters to the script in order to configure the correct credential, the JSON file, and the output folder. 

Figure 16. Jupyter Notebook – Authentication 

Because we registered an Azure Application and we used the application secret to receive an access token, the token is valid for 1 hour. Within the code verify if we need to renew this token before submitting the query. 

Figure 17. Application Token lifetime validation 

When building such flow we should take into consideration Microsoft 365 Defender Advanced hunting API quotas and resources allocation. For more information, see Advanced Hunting API | Microsoft Docs.  

Figure 18. API quotas and resources allocation taking into consideration 

 We run the code by loading the query from the JSON file we defined as input. We then view the progress and the execution status on screen. 

Figure 19. Query Execution 

The blue message indicates the number of queries that is currently running and its progress. 

The green message shows the name of the query that is being run. 

The grey message shows the details of the submitted query. 

If there are any results you will see the first 5 records, and then all the records will be saved in a .csv file in the output folder you defined. 

Figure 20.  Query results – First 5 records 

Bonus 

You can post the summary of the query execution in a Teams channel, you need to add Incoming Webhook in your teams. 

Figure 21.  Incoming Webhook 

Then you need to select which Teams channel you want to add the app. 

Figure 22.  Incoming Webhook – add to a team 

Select “Set up a connector”. 

Figure 23.  Incoming Webhook – Setup a connector 

Specify a name. 

Figure 24.  Incoming Webhook – Config 

Now you need to copy the URL, then paste the URL in the Jupyter Notebook. 

Figure 25.  Incoming Webhook – teamurl variable 

Then remove the comment from the latest line in the code to send the message to Teams. 

Figure 26.  Incoming Webhook – teamsurl variable 

You should receive a similar message like the following in the Teams channel: 

Figure 27.  Query result summary – Teams Message 

Conclusion 

In this post, we demonstrated how you can use the Microsoft 365 Defender APIs and Jupyter Notebook to automate execution of hunting queries playbook. We hope you found this helpful! 

Appendix  

For more information about Microsoft 365 Defender APIs and the features discussed in this article, please read: 

• Access the Microsoft 365 Defender APIs | Microsoft Docs 


• Advanced hunting APIs – Microsoft 365 security | Microsoft Docs 

• Advanced hunting best practices 


• Microsoft-365-Defender-Hunting-Queries 


• Best practices for leveraging Microsoft 365 Defender API’s – Episode Two – Microsoft Tech Community 


• Sending messages to Connectors and Webhooks – Teams | Microsoft Docs 


• Automating Security Operations Using Windows Defender ATP APIs with Python and Jupyter Notebooks – Microsoft Tech Community 

The sample Notebook discussed in the post is available in the github repository
Microsoft-365-Defender-Hunting-Queries/M365D APIs ep3.ipynb at master · microsoft/Microsoft-365-Defender-Hunting-Queries (github.com)

As always, we’d love to know what you think. Leave us feedback directly on Microsoft 365 security center or start a discussion in Microsoft 365 Defender community

April identity updates – Preview of embed Azure AD B2C sign-in interface in an iframe

by Contributed | Apr 26, 2021 | Technology

This article is contributed. See the original author and article here.

Howdy folks,

I’m excited to share the latest Active Azure Directory news, including feature updates, support depreciation, and the general availability of new features that will streamline administrator, developer, and end user experiences. These new features and feature updates show our commitment to simplifying identity and access management, while also enhancing the kinds of customization and controls our customers need.

New features

• Embed Azure AD B2C sign-in interface in an iframe (Preview): Customers have told us how jarring it is to do a full-page redirect when users authenticate. Using a custom policy, you can now embed the Azure AD B2C experience within an iframe so that it appears seamlessly within your web application. Learn more in the documentation.

• Custom email verification for Azure AD B2C (GA): You can send customized email to users that sign up to use your customer applications, with a third-party email provider such as Mailjet or SendGrid. Using a Azure AD B2C custom policy, you can set up an email template, From: address, and subject, as well as support localization and custom one-time password (OTP) settings. Learn more in the documentation.

• Additional service and client support for Continuous Access Evaluation (CAE) – MS Graph service & OneDrive clients on all platforms (Windows, Web, Mac, iOS, and Android) start to support CAE at the beginning of April. Now OneDrive client access can be terminated immediately right after security events, like session revocation or password reset, if you have CAE enabled in your tenant.

We’re always looking to improve Azure AD in ways that benefit IT and end users. Often, these updates originate with the suggestions of users of the solution. We’d love to hear your feedback or suggestions for new features or feature updates in the comments or on Twitter (@AzureAD).

Alex Simons (@Alex_A_Simons)

Corporate VP of Program Management

Microsoft Identity Division

Learn more about Microsoft identity:

• Return to the Azure Active Directory Identity blog home


• Join the conversation on Twitter and LinkedIn


• Share product suggestions on the Azure Feedback Forum

MSTICPy and Jupyter Notebooks in Azure Sentinel, an update

by Contributed | Apr 26, 2021 | Technology

This article is contributed. See the original author and article here.

We published an overview of MSTICPy 18 months ago and a lot has happened since then with many changes and new features. We recently released 1.0.0 of the package (it’s fashionable in Python circles to hang around in “beta” for several years) and thought that it was time to update the aging Overview article.

What is MSTICPy?

MSTICPy is a package of Python tools for security analysts to assist them in investigations and threat hunting, and is primarily designed for use in Jupyter notebooks. If you’ve not used notebooks for security analysis before we’ve put together a guide on why you should.

The goals of MSTICPy are to:

1. Simplify the process of creating and using notebooks for security analysis by providing building-blocks of key functionality.


2. Improve the usability of notebooks by reducing the amount of code needed in notebooks.


3. Make the functionality open and available to all, to both use and contribute to.

MSTICPy is organized into several functional areas:

• Data Acquisition – is all about getting security data into the notebook. It includes data providers and pre-built queries that allow easy access to several security data stores including Azure Sentinel, Microsoft Defender, Splunk and Microsoft Graph. There are also modules that deal with saving and retrieving files from Azure blob storage and uploading data to Azure Sentinel and Splunk.


• Data Enrichment – focuses on components such as threat intelligence and geo-location lookups that provide additional context to events found in the data. It also includes Azure APIs to retrieve details about Azure resources such as virtual machines and subscriptions.


• Data Analysis – packages here focus on more advanced data processing: clustering, time series analysis, anomaly identification, base64 decoding and Indicator of Compromise (IoC) pattern extraction. Another component that we include here but really spans all of the first three categories is pivot functions – these give access to many MSTICPy functions via entities (for example, all IP address related functions are accessible as methods of the IpAddress entity class.)


• Visualization – this includes components to visualize data or results of analyses such as: event timelines, process trees, mapping, morph charts, and time series visualization. Also included under this heading are a large number of notebook widgets that help speed up or simplify tasks such as setting query date ranges and picking items from a list. Also included here are a number of browsers for data (like the threat intel browser) or to help you navigate internal functionality (like the query and pivot function browsers).

There are also some additional benefits that come from packaging these tools in MSTICPy:

• The code is easier to test when in standalone modules, so they are more robust.


• The code is easier to document, and the functionality is more discoverable than having to copy and paste from other notebooks.


• The code can be used in other Python contexts – in applications and scripts.

Companion Notebook

Like many of our blog articles, this one has a companion notebook. This is the source of the examples in the article and you can download and run the notebook for yourself. The notebook has some additional sections that are not covered in the article.

The notebook is available here.

Documentation and Resources

Since the original Overview article we have invested a lot of time in improving and expanding the documentation – see msticpy ReadTheDocs. There are still some gaps but most of the package functionality has detailed user guidance as well as the API docs. We do also try to document our code well so that even the API documents are often informative enough to work things out (if you find examples where this isn’t the case, please let us know). 

In most cases we also have example notebooks providing an interactive illustration of the use of a feature (these often mirror the user guides since this is how we write most of the documentation). They are often a good source of starting code for projects. These notebooks are on our GitHub repo.

Getting Started Guides

If you are new to MSTICPy and use Azure Sentinel the first place to go is the Use Notebooks with Azure Sentinel document. This will introduce you the the Azure Sentinel user interface around notebooks and walk you through process of setting up an Azure Machine Learning (AML) workspace (which is, by default, where Azure Sentinel notebooks run). One note here – when you get to the Notebooks tab in the Azure Sentinel portal, you need to hit the Save notebook button to save an instance of one of the template notebooks. You can then launch the notebook in the AML notebooks environment.

The next place to visit is our Getting Started for Azure Sentinel notebook. This covers some basic introductory notebook material as well as essential configuration. More advanced configuration is covered in Configuring Notebook Environment notebook – this covers configuration settings  in more detail and includes a section on setting up a Python environment locally to run your notebooks.

Although this article is aimed primarily at Azure Sentinel users, you can use MSTICPy with other data sources (e.g. Splunk or anything you can get into a pandas DataFrame) and in any Jupyter notebook environment. The Azure Sentinel notebooks can be found in our Notebooks GitHub repo.

Notebook Initialization

Assuming that you have a blank notebook running (in either AML or elsewhere) what do you do next?

Most of our notebooks include a more-or-less identical setup sections at the beginning. These do three things:

1. Checks the Python and MSTICPy versions and updates the latter if needed.


2. Imports MSTICPy components.


3. Loads and authenticates a query provider to be able to start querying data.

If you see warnings in the output from the cell about configuration sections missing you should revisit the previous Getting Started Guides section. This cell includes the first two functions in the list above. The first one – running utils.check_versions() – is not essential in most cases once you have your environment up and running but it does do a few useful tweaks to the notebook environment, especially if you are running in AML.

The init_notebook function automates a lot of import statements and checks to see that the configuration looks healthy.

The third part of the initialization loads the Azure Sentinel data provider (which is the interface to query data) and authenticates to your Azure Sentinel workspace. Most data providers will require authentication.

Assuming you have your configuration set up correctly, this will usually take you through the authentication sequence, including any two-factor authentication required.

Data Queries

Once this setup is complete, we’re at the stage where we can start doing interesting things!

MSTICPy has many pre-defined queries for Azure Sentinel (as well as for other providers). You can choose to run one of these predefined queries or write your own. This list of queries documented here is usually up-to-date but the code itself is the real authority (since we add new queries frequently). The easiest way to see the available queries is with the query browser. This shows the queries grouped by category and lets you view usage/parameter information for each query.

 

There is also a command-line equivalent function to the browser – qry_prov.list_queries().

 

Timespans

Nearly all queries need time range parameters. You can specify these as parameters to the query function but who wants to type long date-time strings? It usually easier to use the QueryTime widget to set your desired time range and just pass it to the query. In the example below we can see how to load the QueryTime widget. You pass the widget itself to the query function, where the start and end values will be inserted into the query before being run.

 

 

One thing you can see from this screen shot is that the data is returned as a pandas DataFrame. pandas is a package that is extremely popular in the data science community. MSTICPy uses it extensively as a universal data interchange format between different components. We’ll see more examples of DataFrames as we move through the article.

 

Although there are a lot of built-in queries, there will always be cases where you need something different. There are a couple of approaches:

1. Most queries will take an optional parameter add_query_items which allows you to tack on (some might say “inject”!) arbitrary KQL to the query.


2. You can write a query from scratch as a string and just execute it.

These options are shown below.

 

 

Visualizing Data

Now that we can get data, let’s do something more interesting with it.

 

Event Timelines

One of the most basic but also most useful visualizations is to project events onto a timeline. You can do this using MSTICPy‘s separate Timeline function or, more conveniently call it directly from a DataFrame using the mp_timeline pandas extension.

 

 

MSTICPy makes extensive use of the interactive graphics of Bokeh. These charts can be panned and zoomed. Each event also has a hover-over tooltip containing summary information about the event (the summary is derived from the source_columns parameter list).

 

Process Tree

A process tree is another common visualization used when investigating endpoint (host) data.

 

Like the timeline, the process tree supports panning, zooming, hover details and has an optional data table viewer (you need to specify show_table=True when calling the function).

 

Alert Viewer

MSTICPy also has a number of special-purpose viewers for things like alerts, where it is often difficult to see the required data in a tabular format.

 

 

This example combines both the timeline viewer and the SelectAlert browser.

 

Data Enrichment with Threat Intelligence, WhoIs and GeoIP

MSTICPy contains many enrichment components for geo-location, ASN/whois, threat intelligence, Azure resource data and others.

 

This example shows calling a method of the IpAddress entity class to get WhoIs information for an IP address.

 

 

Although the whois feature is available as a standalone function, we’ve used a pivot function of the IpAddress class here. To do this we needed to load the Pivot class.

Side note – Pivot functions

If what you want to do is entity-related, there is a good chance that the MSTICPy function will appear as an entity pivot function. Queries, enrichment functions and analysis functions that relate to a particular entity type are all exposed as pivot functions of that entity.

 

Wait – what is an Entity?

 

An entity is essentially a “noun” in the CyberSec world – for example: an IP Address, host, URL, account, etc. They are typically things that do stuff or have stuff done to them. Entities will always have one or more properties that identify the entity and might have additional context properties. For example, an IpAddress entity has its primary Address property and it might also have contextual properties like geo-location or ASN data.

 

Pivot functions are “verbs” to the entities “nouns”. They perform investigative actions (like data queries) on the entity and return a result. The Host entity class, for example, has data queries that retrieve process or logon events logged for that host. The IpAddress entity has functions to lookup its geolocation or query information about the address from threat intelligence providers.

 

Pivot functions are not statically coded into the entity classes. Instead, the pivot subsystem harvests pivot functions from available queries and components and dynamically adds them to the entities.  This gives us a lot of flexibility to add new functions as the features of MSTICPy evolve. It also allows you to use functions from third party libraries or write your own functions and expose them as pivot functions.

 

How do you find what pivot functions (and even what entities) are available? The easiest way to view the entities, their pivot functions and the help associated with each function is to use the Pivot browser (similar to the query browser shown earlier)

 

 

Being grouped with their respective entities makes the pivot functions easy find (compared with hunting through documents to find the right module or function to import). Pivot functions are grouped into related sub-containers of the entity (so all AzureSentinel queries have the form entity.AzureSentinel.query_function().

 

Another advantage of pivot functions (over standalone functions) is that they have a homogenous interface. They will all accept inputs as single values, lists of values or values stored in DataFrames. They also always return their results as DataFrames.

 

Back to Enrichment

A nice side benefit of pivot functions using DataFrames as both input and output is that we can chain several together in a pandas pipeline. Here we’re taking IP addresses from an alert and successively getting WhoIs data, geo-location data. Finally, we’re querying multiple Threat Intelligence providers to see if they have any data about the IP address. At each stage we’re asking for the new data obtained by each stage to be joined to the previous stage (via the join parameter) – although joining is optional.

 

 

We then display the results received from the threat intel providers in another special-purpose viewer – the TI browser.

 

 

Here is an example of a more manual pipeline that stitches together the base64 decoder, IoC pattern extractor and threat intel lookup. It’s taking an obfuscated PowerShell command line and extracting and examining the contents found in the decoded string. Finally, it displays the TI results in the TI browser.

 

 

We might also want to know where this IP address is physically located. We can do that by using our geo-location enrichment and displaying the results on a map.

 

Using advanced analysis (AKA simple machine learning)

MSTICPy has several more-advanced analysis components that help with identifying anomalous patterns in large data sets. 

 

First we’ll show the time series decomposition and visualization component. This works by determining regular patterns of bulk events (think of outbound network byte counts or logon failures) and then identifies outliers to this pattern with a simple statistical calculation.

 

 

You can see the daily cadence network traffic and the presence of two outlying events on 7-11 (the date, not the store).

 

Where the data is more complex than bulk counts, we can use another anomaly identification technique using Markov Chains. Markov chains is a technique used to predict the future probability of something occurring given the probability of things that happened previously. 

 

Here we are using it to analyze Office 365 data. We will build a model to determine the probability of specific sequences of actions and then identify rare sequences that deviate from this base probability.  Office activity data is first grouped into sessions based on user name and source IP address. The probability of a given sequence of actions within a session is measured during the creation of the model. For example, it would be very common to see actions like opening emails, reading a few files, etc. A session that reads hundreds of files or performs unusual actions like setting a mail forwarding rule or delegating control of a mailbox would be much less probable and so would detectable.

 

(the full code for this is given in the notebook)

 

 

There is clearly one session that stands out from the crowd in terms of the unusual actions seen in that session. We can use the rarity score (the inverse of the probability) to quickly filter out the other sessions and see what was happening. In this case, the event revealed a series of unexpected privilege assignment actions.

 

 

Conclusion

Thanks for sticking with me in through this marathon article. I hope it has given you a flavor of the power of Jupyter notebooks in CyberSec hunting and investigation tasks and a reasonable overview of many of the capabilities in MSTICPy.

 

Take-aways and actions.

• The very obvious first action is go and start playing with notebooks – whether in Azure Sentinel/Azure ML or on your own computer.


• Second would be install MSTICPy and kick the tires. We are always looking for feedback on what does and doesn’t work and are always open to requests or suggestions for new features.


• If you’re feeling adventurous, consider contributing to MSTICPy. These could be ideas that you have that you think would be helpful to the CyberSec community. If you’re a bit stuck for ideas but love security and Python coding, we have a few ideas of our own and way too few people to implement them.


• Read the docs here.


• File an issue, feature request, create a PR or just poke around the code on our GitHub repo.


• Read a summary of the latest release.


• Follow me (@ianhelle), Pete (@MSSPete) and Ashwin (@ashwinpatil) on Twitter.

 

New transactable offers from LANSA, Nuvento, and TransientX in Azure Marketplace

by Contributed | Apr 26, 2021 | Technology

This article is contributed. See the original author and article here.

Microsoft partners like LANSA, Nuvento, and TransientX deliver transact-capable offers, which allow you to purchase directly from Azure Marketplace. Learn about these offers below: LANSA Scalable License: This offer from LANSA provides a Microsoft Windows Server image to use with a Microsoft Azure Resource Manager template in constructing a production-ready Windows stack to deliver LANSA web, mobile, and desktop capabilities. LANSA accelerates development and enables digital transformation. Users can deploy an app to this production environment from any Visual LANSA IDE. NuOCR – OCR automation: This paper-to-digital solution from Nuvento uses optical character recognition (OCR) to automate data extraction. Scanned forms, invoices, surveys, and other documents can be uploaded to a database or to a Microsoft Excel sheet, making them searchable and editable. NuOCR comes with a prebuilt template library for healthcare, insurance, and other industries. TransientAccess Connector: TransientX’s TransientAccess is a zero-trust network access solution that offers an alternative to VPNs. Instead of connecting device to device, TransientAccess uses a temporary hidden network to connect apps on a remote user’s device to apps in a corporate network, dramatically reducing the attack surface. The Connector and the Client are essential components of TransientAccess.

Configuring BitLocker encryption with Endpoint security

by Contributed | Apr 26, 2021 | Technology

This article is contributed. See the original author and article here.

By Luke Ramsdale – Service Engineer | Microsoft Endpoint Manager – Intune

Administrators often work with a variety of devices—newer devices equipped with the trusted platform module (TPM) or older devices and virtual machines (VMs) without TPM. When you’re deploying BitLocker settings through Microsoft Endpoint Manager – Microsoft Intune, different BitLocker encryption configuration scenarios require specific settings. In this final post in our series on troubleshooting BitLocker using Intune, we’ll outline recommended settings for the following scenarios:

• Enabling silent encryption. There is no user interaction when enabling BitLocker on a device in this scenario.


• Enabling BitLocker and allowing user interaction on a device with or without TPM.

As we described in our first post, Enabling BitLocker with Microsoft Endpoint Manager – Microsoft Intune, a best practice for deploying BitLocker settings is to configure a disk encryption policy for endpoint security in Intune.

Enabling silent encryption

Consider the following best practices when configuring silent encryption on a Windows 10 device.

• First, ensure that the Hide prompt about third-party encryption setting is set to Yes. This is important because there should be no user interaction to complete the encryption silently.
  Hide prompt about third-party encryption settings
  
  


• It’s important not to target devices that are using third-party encryption. Enabling BitLocker on those devices can render them unusable and result in data loss.


• If your users are not local administrators on the devices, you will need to configure the Allow standard users to enable encryption during autopilot setting so that encryption can be initiated for users without administrative rights.


• The policy cannot have settings configured that will require user interaction.

BitLocker settings that prevent silent encryption

In the following example, the Compatible TPM startup PIN, Compatible TPM startup key and Compatible TPM startup key and PIN options are set to Blocked. BitLocker cannot silently encrypt the device because these settings require user interaction.

Figure 1. BitLocker – OS Drive Settings

Be aware that configuring these options to Required will also prevent silent encryption. The Required setting involves end user interaction, which is not compatible with silent encryption.

Figure 2. BitLocker – OS Drive Settings

Note
When assigning a silent encryption policy, the targeted devices must have a TPM. Silent encryption does not work on devices where the TPM is missing or not enabled.

Enabling BitLocker and allowing user interaction on a device

For scenarios where you don’t want to enable silent encryption and would rather let the user drive the encryption process, there are several configuration settings that you can use.

Note
For non-silent enablement of BitLocker, the user must be a local administrator to complete the BitLocker setup wizard.

If a device does not have a TPM and you want to configure start-up authentication, set Hide prompt about third-party encryption to Not configured in Base Settings. This will ensure the user is prompted with a notification when the BitLocker policy arrives on the device.

Example setting to configure start-up authentication

If you want to encrypt devices without a TPM, set Disable BitLocker on devices where TPM is incompatible to Not configured. This setting is part of the startup authentication settings and Start-up authentication required must be set to Yes.

Example to encrypt devices without a TPM

If you also want to allow users to configure startup authentication, then set the Startup authentication required setting to Yes and individual settings Allowed, as shown in the example above. These settings will offer all the startup authentication options to the end user.

Managing authentication settings conflicts

If you configure conflicting startup authentication setting options, the device will not be encrypted, and the end user will receive the following message:

BitLocker Drive Encryption error when BitLocker startup options are in conflict.

Or this message:

BitLocker Drive Encryption error when conflicting Group Policy settings are present.

You can use one of the following options to correct the error:

• Configure all the compatible TPM settings to Allowed.
  Compatible TPM settings set to Allowed.
  
  


• Configure Compatible TPM startup to Required and the remaining three settings
  to Blocked.
  Compatible TPM startup to Required and the remaining three settings to Blocked.
  
  


• Configure Compatible TPM startup key to Required and the remaining three settings
  to Blocked.
  Compatible TPM startup key to Required and the remaining three settings to Blocked.
  
  


• Finally, you can configure Compatible TPM startup PIN to Required and the remaining settings to Blocked.
  Compatible TPM startup PIN to Required and the remaining settings to Blocked.

Note: Devices that pass Hardware Security Testability Specification (HSTI) validation or Modern Standby devices will not be able to configure a startup PIN. Users must manually configure the PIN.

If you configure the Compatible startup key and PIN option to Required and all others to Blocked, you will see the following error when initiating the encryption from the device.

Example BitLocker Drive Encryption error when initiating the encryption from the device.

If you want to require the use of a startup PIN and a USB flash drive, you must configure BitLocker settings using the manage-bde command-line tool instead of the BitLocker Drive Encryption setup wizard.

The user role in encryption

After the device picks up the BitLocker policy, a notification will prompt the user to confirm that there is no third-party encryption software installed.

User experience to start BitLocker encryption.

When the user selects Yes, the BitLocker Drive Encryption wizard is launched, and the user is presented with the following start-up options:

User experience to start encryption from the BitLocker Drive Encryption wizard.

Note
If you decide not to allow users to configure start-up authentication and configure the settings to Blocked, then this screen will not be displayed.

If there is no TPM on the device, the user will see the following screen instead of start-up options. The user enters a password when the device restarts.

User experience when there is no TPM on the device.

The user will be presented with recovery key settings. (The options listed will depend on how the recovery key settings have been configured. For more information about recovery key settings, check out the blog on this topic.)

User experience to backup a BitLocker key in the BitLocker Drive Encryption wizard.

Once the recovery key has been saved, the user is prompted to encrypt the device and the process completes.

User experience to encrypt the device in the BitLocker Drive Encryption wizard.

Tip
If you’re unsure about a configuration setting, you can hover the tool tip to see a description and link to additional information:

Hovering over the tool tip to see a description and link to additional information about a BitLocker setting in the Microsoft Endpoint Manager admin center.

Summary

• It is possible to encrypt a device silently or enable a user to configure settings manually using an Intune BitLocker encryption policy.
  The user driven encryption requires the end users to have local administrative rights.


• Silent encryption requires a TPM on the device.


• Be careful when configuring the start-up authentication settings, conflicting settings will prevent BitLocker from encrypting and produce the Group Policy conflict errors.


• For devices without a TPM, set the Disable BitLocker on devices where TPM is incompatible option to Not configured.

More info and feedback

For further resources on this subject, please see the links below.
Enforcing BitLocker policies by using Intune known issues
Overview of BitLocker Device Encryption in Windows 10
BitLocker Group Policy settings (Windows 10)
BitLocker Use BitLocker Drive Encryption Tools to manage BitLocker (Windows 10)

This is the last post in this series. Catch up on the other blogs:

• Enabling BitLocker with Microsoft Endpoint Manager – Microsoft Intune


• Troubleshooting BitLocker from the Microsoft Endpoint Manager admin center


• Troubleshooting BitLocker policies from the client side


• Using BitLocker recovery keys with Microsoft Endpoint Manager – Microsoft Intune

Let us know if you have any additional questions by replying to this post or reaching out to @IntuneSuppTeam on Twitter.

Build and ship software better with GitHub and Microsoft Teams

by Contributed | Apr 26, 2021 | Technology

This article is contributed. See the original author and article here.

Millions of developers use GitHub daily to build, ship, and maintain their software – but software development isn’t performed in silos and require open communication and collaboration across teams. Microsoft Teams is one of the key tools that developers use for their collaboration needs, so it was important that the integration of our platforms provide a seamless, connected experience. Context switching is a drain on productivity and stifles the flow of work, which is why the GitHub integration in Teams is so important – as it gives you and your team full line of sight into your GitHub projects without having to leave Teams. Now, when you collaborate with your team, you can stay engaged in your channels and ideate, triage issues, and work together to move your projects forward.

GitHub has made tremendous updates refreshing the integration – with public preview last September and recently with general availability of the app last month. Many in the developer community have been looking forward to the updates in the new GitHub app, which they’ve experienced on other collaboration platforms, and so we’re excited to share some of new features and the existing capabilities you can experience today.

New support for personal app and scheduling reminders
Since the public preview launch of the GitHub app last September, we’ve made some great updates on a couple key areas. First, we’ve added support for the personal app experience and secondly, we’ve added capabilities to support scheduling reminders for pending pull requests.

Personal app experience
As part of personal app experience, you can now subscribe to your repositories and receive notifications for:

• issues


• pull requests


• commits

All the commands available in your channel are now available on your personal app experience with the GitHub app.

Image showing subscription experience within the personal app view

Schedule reminders
You can now schedule reminders for pending pull requests. With this feature you can now get periodic reminders of pending pull requests as part of your channel or personal chat. Scheduled reminders ensure your teammates are unblocking your workflows by providing reviews on your pull request. This will have an impact on business metrics like time-to-release for features or bug fixes.

Image showing user setting up scheduled reminders within the GitHub Teams app

From your Teams channel, you can run the following command to configure a reminder for pending pull requests on your Organization or Repository:

@github schedule organization/repository

This will create reminder for weekdays at 9.30 AM. However, if you want to configure the reminder for a different day or time, you can achieve that by passing day and time with the below command:

@github schedule organization/repository <Day format> <Time format>

Learn more about channel reminders.

You can also configure personal reminders in your personal chats, as well, using the below command:

@github schedule organization

Stay notified on updates that matter to you through subscriptions
Subscriptions allow you to customize the notifications you receive from the GitHub app for pull request and issues. You can use filters to create subscriptions that are helpful for your projects, without the noise of non-relevant updates – and you can create these in channels or in the personal app.

Subscribing and Unsubscribing
You can subscribe to get notifications for pull requests and issues for an Organization or Repository’s activity.

@github subscribe <organization>/<repository>

Image showing user subscribing to a specific organization and repository via GitHub app

To unsubscribe to notifications from a repository, use:

@github unsubscribe <organization>/<repository>

Customize notifications
You can customize your notifications by subscribing to activity that is relevant to your Teams channel and unsubscribing from activity that is less helpful to your project.

@github subscribe owner/repo

Learn more about subscription notifications on the GitHub app.

Use threading to synchronize comments between Teams and GitHub
Notifications for any pull request and issue are grouped under a parent card as replies. The parent card always shows the latest status of the pull request/issue along with other meta-data like description, assignees, reviewers, labels, and checks. Threading gives context and helps improve collaboration in the channel.

Image showing pull request card staying updated with latest comments and information

Make your conversations actionable
Stay in the flow of work by making the conversations you have with your teammates on GitHub actionable. You can use the app to create an issue, close or reopen an issue, or leave comments on issues and pull requests.

With the GitHub app, you can perform the following:

• Create issue


• Close an issue


• Reopen and issue


• Comment on issue


• Comment on pull request

You can perform these actions directly from the notification card in the channel by clicking on the buttons available in the parent card.

Image of a GitHub pull request notification on a card and the user from Teams commenting

More to come!
We’re excited for you to use these new and existing features as you continue to build world-class software and services – and we’re equally excited to see the growth and evolution of the app in the future. If you haven’t already installed the GitHub app for Teams, you can easily do so today to get started.

Happy coding!