What's new in the Azure Monitor community repo

by Contributed | Nov 4, 2020 | Technology

This article is contributed. See the original author and article here.

With a lot of content added to the community repo, here are some of the highlights:

Azure Inventory Workbook (based on Azure Resource Graph)

This workspace, contributed by Billy York, provides a comprehensive review of the Azure resources you have, arranged by categories (Compute, PaaS, Networking etc.) so very easy to browse through, and details as well.  

AntiMalware Assessment Workbook

This workbook, contributed by Bruno Gabrielli, helps you review the protection status of your resources – which resources are protected and which are not, what protection you use on each and if there are issues (e.g. expired certificates).

Queries

Examples queries from the product were added to many Azure services and solutions, such as Azure AD Domain Services, Virtual Network Gateways, Change Tracking, Container Insights, Security Insight and more.

For the full list of updates visit what’s new on our Azure Monitor community repo.

Let us know what you think – leave your comments here or on our GitHub issues section and share what your liked and what you’d like to see next. 

Microsoft Endpoint Manager: Enable Endpoint Protection

by Contributed | Nov 4, 2020 | Technology

This article is contributed. See the original author and article here.

Introduction

This is John Barbare and I am a Sr. Customer Engineer at Microsoft focusing on all things in the Cybersecurity space. In this tutorial I will walk you through the steps of configuring and enabling Microsoft Defender for Endpoint in Microsoft Endpoint Manager (MEM). The name has been rebranded from Microsoft Defender ATP to Microsoft Defender for Endpoint but inside the portal(s) the name has not changed as of this publication. For this demo, I will show you how to configure almost all settings in the MEM console with the exception of making sure Intune is set to on in Microsoft Defender for Endpoint so all systems are properly synchronized. With that said, let’s get started and see MEM’s awesome capability of the Endpoint Security tab and start protecting your endpoints. For licensing information visit the product licensing terms and Microsoft 365.

Enable Microsoft Defender for Endpoint

The first item you want to do is login into  Microsoft Endpoint Manager admin center and select the Endpoint Security tab on the main column and then under Setup, select Microsoft Defender ATP. Under Common Share Settings select Open the Microsoft Defender ATP admin console.

Setting up Microsoft Defender for Endpoint in MEM

This will take you straight to Microsoft Defender Security Center. On the left hand side select Settings, and then Advanced Features, and make sure Microsoft Intune connection is toggled to the on position and select Apply to save the settings.

Turning on “Intune” in Microsoft Defender for Endpoint

When you established the connection between Intune and Microsoft Defender for Endpoint, Intune received a Microsoft Defender for Endpoint onboarding configuration package from Microsoft Defender for Endpoint. You deploy this configuration package to your Windows devices with a device configuration profile for Microsoft Defender for Endpoint.

Head back over to MEM and under the same Microsoft Defender ATP settings we will configure the MDM Compliance Policy Settings. You have three choices to choose from depending on your organization’s specific security requirements. You can choose from the following:

• Set Connect Windows devices version 10.0.15063 and above to Microsoft Defender ATP to On


• Set Connect Android devices of version 6.0.0 and above to Microsoft Defender ATP to On


• Block unsupported OS versions

If you are only connecting to a Windows device, it is only necessary to turn on the “Set Connect Windows devices version 10.0.15063 and above to Microsoft Defender ATP to On” to the on position. Once you have your organization’s configurations set to on, go ahead and select save.

MEM Compliance Policy Settings

As you can see, the connector settings have been saved successfully as seen in the top right after selecting save with a green check mark.

Configuration Settings Saved

After setting your organization’s configurations to On in the above steps, other applicable devices which are already being managed by Intune, as well as new devices you enroll in the future will be automatically connected to Microsoft Defender for Endpoint for the intent of compliance.

Onboarding a Windows 10 2004 Machine

For this demo, I will be onboarding Microsoft Defender for Endpoint to a new Windows 10 version 2004 machine using MEM.

The configuration package configures devices to communicate with Microsoft Defender for Endpoint services to scan files, detect threats, and other security related items.

One of the best features about the MEM and Microsoft Defender for Endpoint synchronization is after your organization onboards a device using the configuration package, you will never have to do it again.

From the MEM admin center, select Devices, Configuration profiles, + Create Profile, and select Windows 10 and later for Platform.

Creating a Profile

Under Profile select Microsoft Defender ATP (Windows 10 Desktop), and select Create.

Adding Defender in the Profile

This will take you to the basics page and put a good name and description of your profile and select next.

Adding in Name and Description

For configuration settings, select the configurations that work best for your company and select next.

Configuration Settings

For Assignments, select either all users, all devices, or all users and devices. If you have groups to exclude, select exclude and add the groups to the list and select next.

Adding Assignment Settings

For Applicability Rules it will give you three options to choose from: Rule, Property, and Value. An example would be to create a rule that has the OS edition of WIN 10 EDU, WIN ENT, WIN 10 PRO , WIN 10 PRO EDU, and WIN 10 PRO WKS. Select next when complete.

Selecting Which O/S Types Receive Policy

Review your new policy that you created that defined all the correct parameters in your policy before selecting create.

Confirming Policy

As seen on the upper right, the MDATP WIN 10 Desktop profile was successfully created.

Profile Created

Confirming The Windows Device Was Successfully Onboarded in MEM

Go to Endpoint Security, then Microsoft Defender ATP and it will show you the device (or your devices) that have been onboarded successfully with the sensor as seen below.

Device(s) Successfully Onboarded

Confirming in Microsoft Defender Security Center

Click on https://securitycenter.windows.com/ and select Configuration Management tab on the bottom. The machines that fit the criteria based on the OS versions in the created policy will show up inside the portal as seen below.

Microsoft Defender for Endpoint Displaying Successfully Onboarded Devices in MEM

To see the machines that were onboarded and other information, one can select the Device Inventory tab and confirm by Machine Name, Health state, and when it was last seen online to confirm.

Quick Video Demo

To see this in action, click here for a video walkthrough from a demo from Microsoft’s Product Group. Enjoy! 

Conclusion

Thanks for taking the time to read this blog and I hope you had fun reading how to onboard WIN 10 devices to Microsoft Defender for Endpoint without downloading or deploying a package as everything is done inside MEM. This configuration that was demoed configures your devices to communicate with Microsoft Defender for Endpoint services to scan files and detect threats. The device is also configured to report to Microsoft Defender for Endpoint the devices risk level based on compliance policies you will create. Hope to see you in the next blog and always protect your endpoints!

Thanks for reading and have a great Cybersecurity day!

Follow my Microsoft Security Blogs: http://aka.ms/JohnBarbare  

Azure Defender for Key Vault

by Contributed | Nov 3, 2020 | Technology

This article is contributed. See the original author and article here.

We are excited to share that Azure Defender for Key Vault has been generally available since Microsoft Ignite on September 22nd, 2020! We have prepared this blog to go over the following topics: 

• Introduction to Azure Defender for Key Vault 


• How to enable Azure Defender for Key Vault 


• How to find alerts of Azure Defender for Key Vault 


• How to respond to alerts 


• How to provide feedback on alerts

Azure Defender for Key Vault

Azure Key Vault is a cloud service for securely storing and accessing secure data like secrets, keys, certificates, and passwords. By compromising this data, attackers may be able to gain unauthorized access or perform lateral movement to breach other resources in the customers’ environment. 

Azure Defender for Key Vault is an Azure-native threat protection service, which detects unusual and potentially harmful access to Key Vault accounts. It provides an additional layer of security intelligence for the keys, secrets and certificates stored in the Azure Key Vault by alerting you to suspicious or malicious access. This layer of security allows you to address threats without being a security expert, and without the need to manage third-party security monitoring systems. 

When anomalous activities occur, Azure Defender shows alerts and optionally sends them via email to relevant members of your organization. These alerts include the details of suspicious activity and recommendations on how to investigate and remediate the threats. 

More information about Azure defender for Key Vault is available here. 

How to enable Azure Defender for Key Vault

Azure Defender for Key Vault can be enabled from Security Center, or from Key Vault. The pricing information is available here. 

Method 1: Enable from Security Center

1. Go to the Security Center from Azure Portal. 


2. From Security Center’s main menu, select Pricing & settings under Management.  
   
    
   
   


3. Select the subscription that you want to enable Azure Defender for Key Vault. 


4. Select Azure Defender to upgrade.  
   

    

   
   


5. Switch Azure Defender plan for Key Vault to On. 


6. Select Save. 

Method 2: Enable from Key Vault 

1. Go to the target Key Vault from Azure Portal. 


2. From Key Vault’s main menu, select Security under Settings.  
   

    

   
   


3. Under Security alerts, select Try it free for the first 30 days.  


4. Select the subscription that you want to enable Azure Defender for Key Vault.   
   

    

   
   


5. Select Upgrade. 

How to find alerts of Azure Defender for Key Vault

The Azure Defender for Key Vault alerts show up on Key Vault and Security Center. 

1. View alerts on the Key Vault’s Security page 
   
   
   
   1. Go to the target Key Vault from Azure Portal.   
   
   
   2. From Key Vault’s main menu, select Security under Settings.   
   
   
   
   


2. View alerts in the Security Center’s Security alerts page. 
   
   
   
   1. Go to the Security Center from Azure Portal.   
   
   
   2. From Security Center’s main menu, select Security alerts under General. 
      

        

      
      
   
   
   
   


3. Details on the alert page.
   

    

   
   

Here is the list of alerts that you might get from the Azure Defender for Key Vault.  

How to respond to alerts

Azure Defender for Key Vault is designed to help identify suspicious activity caused by stolen credentials. Do not dismiss the alert simply because you recognize the user or application. Contact the owner of the application or the user and verify the activity was legitimate. 

When you get an alert from Azure Defender for Key Vault, we recommend following this document.

How to provide feedback on alerts

Please provide your feedback for each alert on the alert page, which provides valuable input for the algorithm developers in the team to improve the quality of the alerts in the future. This feedback will not directly affect the results of the algorithm and will only be used to make long-term improvements. 

1. Select the answer to the question Was this useful?. 


2. Choose the Reason which best matches your scenario. 


3. Provide Additional feedback to help the service team understand more about the reason why it is useful or not. 


4. Select Microsoft may email me about my feedback to allow the service team to follow up by email. 


5. Select Submit. 

You can create alert suppression rules to suppress unwanted security alerts from Azure Defender. Learn more in Suppress alerts from Azure Defender. 

Clickstream analytics in Azure Data Explorer

by Contributed | Nov 3, 2020 | Technology

This article is contributed. See the original author and article here.

Clickstream analysis is the process of collecting, analysing and reporting aggregated data about user’s journey on a website. User’s interactions with websites are collected, with applications like Adobe analytics and Tealium. Bringing clickstream data into a centralised place and combining it with other data sources for rich analytics is often required.

Typically clickstream data can reach around 2-3TB/day in size. Using Relational databases to analyse such data might not be suitable or cost effective.

Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of Telemetry, Logs, Time Series data streaming from applications, websites, IoT devices, and more.

Proposed Architecture:

An end to end proposed architecture how you can use Azure Data Explorer to bring Clickstream historical and incremental data into Azure Data Explorer and analyse, aggregate and visualise the data.

Source System:

1.a) Adobe Analytics dataset usually contains about 1000+ fields. Adobe Analytics provides Data feed functionality which can be used to extract the data to Azure Data Lake Store. Data can be extracted with different configurations:

1. Frequency: daily or hourly


2. Format: .zip or .gz format


3. Files: Single vs Multiple

1.b) Tealium datasets are extracted in JSON structure with 100’s of nested fields. The data can be sent to Azure Event hub in near realtime.

Bringing Data into ADX: 

Creating the table structure and mapping for 1000+ column is a cumbersome task, it can be made easy with 1 Click Ingestion.

There are multiple ways to load the data.

3) Historical Load using Lightingest:

LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer.

Simplest and efficient way to load all the historical data, with just one command.

LightIngest.exe "https://ingest-<<ClusterName.ClusterRegion>>.kusto.windows.net;Fed=True"   -database:<<DatabaseName>>   -table:<<TableName>>   -source:" https://<<StorageName>>.blob.core.windows.net/<<ContainerName>>;<<StorageAccountKey>> "   -prefix:"<<FolderPath>>"   -pattern:*.tsv.gz   -format:tsve

Although adobe files are in TSV format but due to special characters in the files, you should use the format TSVE in ADX.

Similarly, for Tealium historical load, use Json format.

Incremental Load:

3.a) 

For Adobe batch file loading subscribe to eventgrid.

Alternatively, ADF is useful for loading because you can easily control the loading flow, and manage all your ETL or ELT pipelines from a single tool. Example pipeline (Template attached)

.ingest command:

.ingest into table @{pipeline().parameters.tableName} (h'abfss://@{variables('containerName')}@{pipeline().parameters.storageName}.dfs.core.windows.net/@{variables('folderPath')}/@{pipeline().parameters.fileName};@{activity('GetAccountKeyFromKV').output.value}') with (format='tsve',ingestIfNotExists = '["@{pipeline().parameters.fileName}"]', tags = '["ingest-by:@{pipeline().parameters.fileName}"]',creationTime='@{variables('inputFileDateTime')}')

3.b) For Tealium real time streaming connect Eventhub to ADX table:

4) Expected Compression ratio:

5) Querying & Analytical workload:

The Analytical work requires more compute but is usually required at working hours. Using a follower cluster will make it easy to pause/resume and optimise for read workload. This will be useful for charge back to different groups and will provide workload isolation.

Cluster pause/resume can be done using Logic app or Azure Automation.

Estimated Cluster Size:

The cluster size will depend on the daily data ingestion and how many days needs to be retained in hot cache. You can estimate your cluster size based on your requirements.

Costing for 2TB daily load with 10x compression.   

You can monitor your workload and adjust your cluster accordingly.

Why Is It So Ridiculously Slow To Remove My Query Resolution Policies!?

by Contributed | Nov 3, 2020 | Technology

This article is contributed. See the original author and article here.

Hi Team, Eric Jansen here – I’m a Platforms Customer Engineer with some interesting scenarios that I’ll be sharing over the coming months, mostly revolving around the topic of On-Premises DNS.  Today’s topic will be a very specific scenario regarding DNS Policy.  DNS Policy has been around since the debut of Windows Server 2016 and it was a massive leap forward in functionality for Windows On-Prem DNS.  There’s plenty of content regarding that topic, so that’s not what I’ll be talking about today, but if you want an overview, check out the following:

https://docs.microsoft.com/en-us/windows-server/networking/dns/deploy/dns-policy-scenario-guide

More specifically, today’s topic is regarding the removal of DNS Query Resolution Policies (or from here forward ‘QRPs’ – ‘Query Resolution Policies’ is a lot of typing ) in large scale.  To provide some context, a number of my customers use DNS as one of their methods for blocking their clients from getting to unwanted domains that may have been identified as malicious or go against corporate policy, or whatever the reason may be.  With that said, there are customers that have hundreds of thousands of these policies to block hundreds of thousands of domains.  So now comes the inevitable question, when they first try to remove all the policies in doing their initial testing with so many policies – Why is it so ridiculously slow to remove my Query Resolution Policies!?

The answer to that is… Well, it depends.  It depends on how you attempted to remove them.

With that said, let’s look at a scenario in the lab using a domain joined member server that has the DNS Role installed, where customer X is trying to remove, let’s say 50,000 QRPs.  Below are some options and we’ll measure the time it takes for each option to complete, but first a Pro Tip:

Pro Tip: Incase this is your first rodeo – Always test in a lab first and have a thorough understanding of what you’re doing before making production changes.  When you do make your changes, in your lab and in production, have a backout plan.

Option 1: 

Get-DNSServerQueryResolutionPolicy | Remove-DNSServerQueryResolutionPolicy -Force

Sounds logical to me…but unfortunately this doesn’t work for all scenarios, especially if you have a very large number of domains.

Eventually, at least in all of my testing, it’ll fail and throw an exception ID of Win32 167.  If you dig into the exception further ($Error[0].Exception.ErrorData) it translates to “Unable to lock a region of a file.”  This can happen due to a timeout thread that does the re-arrangement of the policies, that can block the addition or creation of policies due to a read lock, when too many changes are being made at once.

Timer:  N/A – Fail..

Yeah, so it sounds like more than one thing is trying to be changed at the same time and it’s not happy.  I know, this is not fun for anyone..  Ok, lets throttle it back a bit – quite literally.

Option 2: 

Get-DNSServerQueryResolutionPolicy | Remove-DNSServerQueryResolutionPolicy -Force -ThrottleLimit 1

OK, so we start to look at additional parameters for Remove-DNSServerQueryResolutionPolicy and find the -ThrottleLimit parameter that shows the following description:

“Specifies the maximum number of concurrent operations that can be established to run the cmdlet. If this parameter is omitted or a value of 0 is entered, then Windows PowerShell® calculates an optimum throttle limit for the cmdlet based on the number of CIM cmdlets that are running on the computer. The throttle limit applies only to the current cmdlet, not to the session or to the computer.”

So, if you have concurrent operations happening, but some operation needs to be changed before another operation can occur, maybe this will help?  Yep, the ThrottleLimit value maintains the order of processing policies into the pipeline, so it essentially serializes the data going through the pipeline (at least in our scenario with the ThrottleLimit value being 1) without the need to collect the data first and then use a foreach loop (as an example), getting all policies and for each one, removing it.

OK, so, let’s see what happens.

Well played…. It now works with no exceptions being thrown!  But the excitement wears off pretty quickly, and you’re learning the hard way that unfortunately, now it takes forever.  So now instead of throwing exceptions you’re ready to throw your keyboard in frustration.   It took 31 hours, 24 minutes and 28 seconds to complete…. That’s no good, so let’s see if we can figure something else out.

Option 3:

Get-DNSServerQueryResolutionPolicy | Sort ProcessingOrder -Descending |

Remove-DNSServerQueryResolutionPolicy -Force

We have now outsmarted the system!  But why is this faster?? 

Well, consider the following.  When you do a Get-DNSServerQueryResolutionPolicy you’ll notice that it returns the list sorting it based on the processing order.  OK, so what happens if I remove the QRP that is assigned to ‘ProcessingOrder’ 1?  Yep, the other 49,999 policies are now moved up in processing order, and then the next policy is removed, which now has ‘ProcessingOrder’ 1, so the remaining 49,998 policies have to have their ‘ProcessingOrder’ modified, and so on, until they’re all gone.  OK, so let’s do the math on that for 50,000 QRPs that need to get removed using ‘Option 2′.

After about six hours of using calc.exe, my fingers started to get tired of entering in the numbers into the calculator:  49,999 + 49,998 + 49,997 + 49,996 + 49,995 etc.;  So I decided that I’d figure it out with PowerShell instead. (Ok, I may have lied – the thought of entering 50,000 values into the calculator never once crossed my mind.)

$ProcessingOrder = 1..50000
$i = 0
$j = 0
Foreach($Order in $ProcessingOrder){
$i++
$($j += $($ProcessingOrder.count - $i))
}
 
Write-Host "Number of Processing Order Changes that will need to take place with 'Option 2':"
Write-Host "$('{0:N0}' -f $j)" -ForegroundColor Red

Running the snippet of code above will return this:

So, if the calculation that my code does above is correct, then…that’s a lot of changes that need to take place, and that is the answer to why it’s so ridiculously slow.

Option 3 on the other hand removes the policies in reverse processing order, so that no other policies need to have the ‘ProcessingOrder’ value modified for them – I’m sure most admins would prefer this methodology, of only having to make 50 thousand changes (just deletes), vs. just shy of 1.25 BILLION processing order changes…plus the 50 thousand deletes.

Now you may have noticed that I did leave the -ThrottleLimit parameter off of Remove-DNSServerQueryResolutionPolicy so it’s technically still removing more than one at a time, and not necessarily in the exact reverse order, as evident from the Audit log.  Option 2, however, was going exactly in ascending order, and it would take on average 5 second to delete each policy in the beginning (based on the audit log timestamps).

Just because it’s you guys though, and I because I know you’re curious, I shimmed in an Option 3.5, just to test the time of doing it in reverse order, but this time setting the throttle limit to 1.

Option 3.5:

Get-DnsServerQueryResolutionPolicy | Sort-Object ProcessingOrder -Descending |

Remove-DnsServerQueryResolutionPolicy -Force -ThrottleLimit 1

Well, it’s not going to win any speed records at the racetrack, but it’s not that bad.  I’ll take this over ‘Option 2′ all day long.

But WHAT IF….we could do one better??  WHAT IF, you really had that need for speed?  Well team, let me introduce you to the high speed, low drag, ‘Option 4′.

Option 4: 

Stop-Service DNS

Get-Item ‘HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionDNS ServerPolicies’ |

Remove-Item -Recurse -Force

Start-Service DNS

From a speed perspective we have a winner…but you have to take the service down, so I’m not a fan of that, but if you have a lot of resolvers, it may not matter to you..

Specs:

For those that are interested, the testing was done on a Windows Server 2019 (Ver 1809) VM that was running on a Windows Server 2019 (Ver 1809) Hyper-V box.  The VM has the following hardware configuration:

The Hyper-V host is an old Dell R610.

Side note:  The test scenarios above were more memory intensive than CPU, I just used an existing DNS server that I had laying around and it just happened to have 24 CPUs already on it .  Regardless of the numbers that I posted for the test scenarios above, everyone’s mileage will vary, and the point of the article wasn’t to give exact numbers, but to show the difference in time that it takes depending on the approach that’s taken to remove the QRPs.

Until next time..

Disclaimer:

The content above contains a sample script.  Sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.