This article is contributed. See the original author and article here.

In the previous episode, we described how you can easily use PowerBi to represent Microsoft 365 data in a visual format. In this episode, we will explore another way you can interact with the Microsoft 365 Defender API. We will describe how to automate data analysis and hunting using Jupyter notebook.


 


Automate your hunting queries 


While hunting and conducting investigations on a specific threat or IOC, you may want to use multiple queries to obtain wider optics on the possible threats or IOCs in your network. You may also want to leverage queries that are used by other hunters and use it as a pivot point to perform deep analysis and find anomalous behaviors. You can find a wide variety of examples in our Git repository where various queries related to the same campaign or attack technique are shared.  


In scenarios such as this, it is sensible to leverage the power of automation to run the queries rather than running individual queries one-by-one.  


This is where Jupyter Notebook is particularly useful. It takes in a JSON file with hunting queries as input and executes all the queries in sequence. The results are saved in a .csv file that you can analyze and share. 


 


Before you begin 


JUPYTER NOTEBOOK 


If you’re not familiar with Jupyter Notebooks, you can start by visiting https://jupyter.org for more information. You can also get an excellent overview on how to use Microsoft 365 APIs with Jupyter Notebook by reading Automating Security Operations Using Windows Defender ATP APIs with Python and Jupyter Notebooks.   


 


VISUAL STUDIO CODE EXTENSION 


If you currently use Visual Studio Code, make sure to check out the Jupyter extension 


msftdario_27-1619422918103.png


Figure 1. Visual Studio Code – Jupyter Notebook extension 


 


Another option to use Jupyter Notebook is the Microsoft Azure Machine Learning service. 


Microsoft Azure Machine Learning is the best way to share your experiment with others and for collaboration. 


Please refer to Azure Machine Learning – ML as a Service | Microsoft Azure for additional details. 


msftdario_28-1619422918116.png


Figure 2. Microsoft Azure Machine Learning 


 


In order to create an instance, create a resource group and add the Machine Learning resource. The resource group lets you control all of the resources from a single entry point. 


msftdario_29-1619422918122.png


Figure 3. Microsoft Azure Machine Learning – Resource 


 


When you’re done, you can run the same Jupyter Notebook you are running locally on your device.  


msftdario_30-1619422918118.png


Figure 4. Microsoft Azure Machine Learning Studio 


 


App Registration 


The easy way to access the API programmatically is to register an app in your tenant and assign the required permissions. This way, you can authenticate using the application ID and application secret. 


Follow these steps to build your custom application. 



msftdario_31-1619422977477.png


Figure 5. App registration 


  


Select “NEW REGISTRATION“. 


  


msftdario_32-1619422977481.png


Figure 6. Register an application 


 


Provide the Name of your app, for example, MicrosoftMTP, and select Register. 


Once done, select “API Permission“. 


  


msftdario_33-1619422977495.png


Figure 7. API Permissions 


  


Select “Add a permission“. 


msftdario_34-1619422977484.png


Figure 8. Add permission 


 


Select the “APIs my organization uses“. 


  


msftdario_35-1619422977485.png


  Figure 9. Alert Status 


  


msftdario_36-1619422977486.png


Figure 10. Request API permission 


  


Search for Microsoft Threat Protection and select it. 


msftdario_37-1619422977487.png


Figure 11. Microsoft Threat Protection API 


 


Select “Application Permission“. 


msftdario_38-1619422977489.png


Figure 12. Application Permissions 


 


Then select: 



  • AdvancedHunting.Read.All 

  • Incident.Read.All 


 


msftdario_39-1619422977491.png


Figure 13. Microsoft 365 Defender API – Read permission 


 


Once done select “Add permissions“. 


msftdario_40-1619422977492.png


Figure 14. Microsoft 365 Defender API – Add permission 


 


Get Started 


Now that we have the application ready to access the API via code, let’s try to see is any of the Qakbot queries shared in Microsoft 365 Defender Git produce any results. 


msftdario_41-1619423114158.png


 


Figure 15. Microsoft 365 Defender – Hunting Queries 


 


The following queries will be used in this tutorial:  


 


Javascript use by Qakbot malware 


Process injection by Qakbot malware 


Registry edits by campaigns using Qakbot malware 


Self-deletion by Qakbot malware 


Outlook email access by campaigns using Qakbot malware 


Browser cookie theft by campaigns using Qakbot malware 


Detect .jse file creation events 


 


We need to grab the queries that we want to submit and populate a JSON file with this formatPlease be sure that you are properly managing the escape character in the JSON file (if you use Visual Studio Code (VSCode) you can find extensions that can make the ESCAPE/UNESCAPE process easiest, just pick your favorite one). 


 


 


 


 


 

[ 
        { 
            "Description": "Find Qakbot overwriting its original binary with calc.exe", 
            "Name": "Replacing Qakbot binary with calc.exe", 
            "Query": "DeviceProcessEvents | where FileName =~ "ping.exe" | where InitiatingProcessFileName =~ "cmd.exe" | where InitiatingProcessCommandLine has "calc.exe" and InitiatingProcessCommandLine has "-n 6" and InitiatingProcessCommandLine has "127.0.0.1" | project ProcessCommandLine, InitiatingProcessCommandLine, InitiatingProcessParentFileName, DeviceId, Timestamp", 
            "Mitre": "T1107 File Deletion", 
            "Source": "MDE" 
        } 
] 

 


 


 


 


 


Once you have all your queries properly filled, we must provide the following parameters to the script in order to configure the correct credential, the JSON file, and the output folder. 


msftdario_42-1619423295313.png


Figure 16. Jupyter Notebook – Authentication 


 


Because we registered an Azure Application and we used the application secret to receive an access token, the token is valid for 1 hour. Within the code verify if we need to renew this token before submitting the query. 


msftdario_43-1619423295303.png


Figure 17. Application Token lifetime validation 


 


When building such flow we should take into consideration Microsoft 365 Defender Advanced hunting API quotas and resources allocation. For more information, see Advanced Hunting API | Microsoft Docs.  


msftdario_44-1619423295312.png


Figure 18. API quotas and resources allocation taking into consideration 


 


 We run the code by loading the query from the JSON file we defined as input. We then view the progress and the execution status on screen. 


msftdario_45-1619423295315.png


Figure 19. Query Execution 


 


The blue message indicates the number of queries that is currently running and its progress. 


The green message shows the name of the query that is being run. 


The grey message shows the details of the submitted query. 


If there are any results you will see the first 5 records, and then all the records will be saved in a .csv file in the output folder you defined. 


 


msftdario_46-1619423295309.png


Figure 20.  Query results – First 5 records 


 


Bonus 


You can post the summary of the query execution in a Teams channel, you need to add Incoming Webhook in your teams. 


 


msftdario_47-1619423368892.png


Figure 21.  Incoming Webhook 


 


Then you need to select which Teams channel you want to add the app. 


msftdario_48-1619423368929.png


Figure 22.  Incoming Webhook – add to a team 


 


Select “Set up a connector”. 


msftdario_49-1619423368932.png


Figure 23.  Incoming Webhook – Setup a connector 


 


Specify a name. 


msftdario_50-1619423368937.png


Figure 24.  Incoming Webhook – Config 


 


Now you need to copy the URL, then paste the URL in the Jupyter Notebook. 


msftdario_51-1619423368906.png


Figure 25.  Incoming Webhook – teamurl variable 


 


Then remove the comment from the latest line in the code to send the message to Teams. 


msftdario_52-1619423368909.png


Figure 26.  Incoming Webhook – teamsurl variable 


 


You should receive a similar message like the following in the Teams channel: 


msftdario_53-1619423368914.png


Figure 27.  Query result summary – Teams Message 


 


Conclusion 


In this post, we demonstrated how you can use the Microsoft 365 Defender APIs and Jupyter Notebook to automate execution of hunting queries playbook. We hope you found this helpful! 


 


Appendix  


For more information about Microsoft 365 Defender APIs and the features discussed in this article, please read: 




The sample Notebook discussed in the post is available in the github repository
Microsoft-365-Defender-Hunting-Queries/M365D APIs ep3.ipynb at master · microsoft/Microsoft-365-Defender-Hunting-Queries (github.com)


 


As always, we’d love to know what you think. Leave us feedback directly on Microsoft 365 security center or start a discussion in Microsoft 365 Defender community

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.