AI powered Azure tools

by Contributed | Jan 20, 2021 | Technology

This article is contributed. See the original author and article here.

If you are a frequent user of Azure CLI or Azure PowerShell, then chances are that you have experienced one of the AI-powered features our service provides for the Azure command line tools. These features include generating up-to-date command usage examples with each new Azure CLI and Azure PowerShell releases to ensure the documentation is always up to date (“–help” or “-h” in the command line), enabling natural language search of a command in the Azure CLI (az find), assisting with failure recovery, and the recently released AZ Predictor (for Azure PowerShell). Our AI service for Azure tools serves thousands of user requests per second. In order to handle such high scale application workloads we use an Azure App Service tier called App Service Environments (ASE). While ensuring our Azure App Service is scalable and performant, we had to address several key challenges like being able to stress test the service with a high volume of traffic, running the service in fully isolated environments without affecting other Azure subscriptions, and throttling bad actors beyond reasonable levels of requests. We will share our journey of discovery, surprises, and learnings that led us to arrive at an enterprise-grade, cloud-scale AI service that powers our own client tools.

Architecture

When a user interacts with one of our integrations in Azure CLI or Azure PowerShell, a request is made to our App Service. To fulfil the request, the App Service gets the result from the relevant AI model. Note that we use different resources for different types of models. For example, the model that generates examples when a user tries to find a command is powered by Azure Search with natural language search features. The failure recovery and AZ Predictor models are more suited to be stored in Azure Storage and loaded into memory to achieve high performance.

Considering the geo-diversity of our users and the high volume of traffic, the service runs in multiple regions to achieve quick response time and high availability. Each request is routed to the nearest region by Azure Traffic Manager (1). In each region, the REST API might talk to:

• Azure Key Vault (2) to retrieve secrets.


• Azure Redis Cache (3) to get or set cached response for improved performance.


• Azure Search (4) to get some of the model results when they are not cached.


• Azure Storage (5) to load some of the models into memory at service startup or when models need to be refreshed.

Figure 1: App Service Architecture

Extensibility and Scalability

As we continue to enable more features in the service, it is important to design the App Service in a scalable way. We develop common libraries for accessing Azure resources like Azure Key Vault, Azure Search, and Azure Redis Cache. Those libraries can even be shared with other services as well.

Scalability and performance are critical for the service. A user typing on the command line typically expects a response in less than a second. This requires the service to respond to requests quickly, in the order of milliseconds. We receive thousands of requests per second (RPS), so we need to make sure the service scales fast enough to meet our changing demand. For example, we might experience a peak in the usage pattern when a new version of the Azure CLI comes out. To configure the scaling properly, we started by analyzing Azure CLI usage patterns to understand the expected traffic trends. Figure 2 shows the hourly traffic for the service in the order of 1000s.

Figure 2: Service Hourly Traffic

Based on the extremes from the usage data, the RPS estimation we used for stress testing was in the order of several 1000s of requests. Note that the estimation we used to stress test is higher than the traffic showed in this figure. This is in part because not all Azure CLI or Azure PowerShell users have upgraded to versions with our service integrations. We decided to play it safe with stress testing and prepare for the maximum number of requests we might receive plus some additional margin.

As it turned out, being able to stress test the service with thousands of requests per second was a huge challenge.

Stress Test Journey

We explored a variety of stress testing tools including loader.io and JMeter. At the end we decided to use JMeter because it offers a few key features: the ability to control requests per second, it is fully supported in Azure DevOps, and it is actively maintained. However, using JMeter involved its own challenges:

• RPS limit from one machine. We quickly found that because of the amount of TCP connections created by our JMeter script, we could only send around 1,000 RPS from one machine.


• Configuration needed to maintain multiple VMs in VMSS. Using multiple machines was necessary to overcome the above RPS limit from one machine.

We then considered running Azure Container Instance (ACI) to overcome these limitations. ACI is Azure CLI-friendly and we found it was quite easy to prepare and run a container in Azure. We were able to easily and quickly launch our images in multiple ACIs to reach our target RPS without the pain of having to individually configure and maintain multiple VMs.

App Service Environments

This led us to an unexpected challenge. Hitting our service with 1000s of RPS brought down our service, because it was overloading the load balancing front ends for that stamp. A stamp is a cluster of servers. You can think of it as groups of racks with shared resources like traffic routing within an Azure Data Center. To avoid this issue, we moved to App Service Environments (ASE) for our infrastructure. ASE provides a fully-isolated and dedicated environment for securely running App Services on a high scale. This includes isolated load balancing front ends for network routing.

With the right architecture in place, we were able to scale both horizontally and vertically. Horizontal scaling is done by increasing the number of routing front ends and App instances in proportion to actual traffic. Vertical scaling is done through choosing the right tiers for Azure Redis Cache, Azure Search Service, and App Service plan.

Throttling

Bad actors and broken scripts could bombard the service beyond reasonable levels of requests. Instead of allowing these clients to potentially bring the service down, we would ideally start blocking individual bad actors. The service implements both Client- and IP-based rate limiting. Having both these options allow us to be flexible, minimize slowdowns, and keeps the service running even when hit by bad actors.

Reliability

As shown in the architecture diagram, we deploy the service to multiple geographic regions which are aligned with the distribution of Azure CLI users to make sure every user’s request can be served from a deployment with low latency. This helps us ensure we meet our service health targets. Our reliability goal is that 99.99% of requests in the preceding 5 minutes were successful. Our latency goal is that for the requests in the preceding 15 minutes are served in under 700 milliseconds for the 99.9th percentile.

All traffic for the services is routed by Azure Traffic Manager. When Traffic Manager detects that a region is unhealthy (which is checked via a monitor API defined in the service) the traffic is then routed to the next closest region.

Deployment

We use Azure DevOps and Azure Resource Manager templates (ARM) to deploy the service. Deployment is first tested in Dev and PPE (pre-production environments) environments. An interesting learning is that the first ASE deployment takes 1.5 – 2 hours, though subsequent deployments are much shorter.

Our release pipeline follows the ring-based deployment process with the goal of being able to limit failures to the minimum number of customers.

Monitoring

Monitoring is important to make sure we know right away when something goes wrong. This is essential for us to meet our reliability and availability requirements. The service logs various metrics which are then used in a variety of alerts to the team: when an API call fails, when the App Service performance is degraded, when some resource in a region fails, when a region goes down etc. The metrics data is processed in near-real-time alerts (60s ingestion latency).

Help us make the tools that you love and use every day even better. You can provide feedback by creating an issue in Azure CLI GitHub page and Azure PowerShell GitHub Page.

Bring Threat Intelligence from IntSights Using TAXII Data Connector

by Contributed | Jan 20, 2021 | Technology

This article is contributed. See the original author and article here.

Azure Sentinel provides multiple ways to import threat intelligence into the product. The threat intelligence imported into Azure Sentinel makes it to the ThreatIntelligenceIndicator log analytics table and is also made available on the Threat Intelligence Blade. The imported threat intelligence can then be used in various parts of the product like hunting, investigation, analytics, workbooks, etc.

One of the ways to bring threat intelligence into Azure Sentinel is using the Threat Intelligence – TAXII Data connectors. This data connector uses the TAXII protocol for sharing data in STIX format and enables a built-in TAXII client in Azure Sentinel to import threat intelligence from TAXII 2.x servers.

Today we are announcing the availability of the IntSights TAXII Server, which allows you to get threat data from IntSights into Azure Sentinel using the Threat Intelligence – TAXII Data connector.

IntSights + Microsoft Azure Sentinel Solution

The IntSights and Azure Sentinel integration allows customers to gain visibility on indicators of compromise that directly target their unique digital footprints based on monitoring of thousands of sources across the clear, deep, and dark web to identify threats. With the IntSights intelligence feed, users can proactively block threats before they impact their environments and get instant visibility on those threats within Azure Sentinel.

In addition, users can leverage this integration to get enriched IOCs with additional context about the IOC like threat actor, malware, and campaign information. Users can further pivot to the IntSights Investigation module to gain additional visibility and context to protect their organizations.

Microsoft Azure Sentinel benefits with IntSights feed

The Azure Sentinel and IntSights integration enables analysts in the Security Operation Center (SOC) to gain visibility into thousands of indicators of compromise with contextual information. This integration also provisions dashboards in the form of Azure Sentinel workbooks detailing event data associated with the IOCs found in your environment.

This blog will walk you through the process of connecting the IntSights TAXII Server to Azure Sentinel.

Connecting Azure Sentinel to the IntSights TAXII Server:

To connect Azure Sentinel to the IntSights TAXII Server, obtain the API Root, Collection ID, Username and Password from the IntSights portal after you configure a policy of the data you wish to send to Azure Sentinel.

Enable the Threat Intelligence – TAXII data connector in Azure Sentinel

To import threat intelligence indicators into Azure Sentinel from the IntSights TAXII Server, follow these steps:

1. Open the Azure Portal and navigate to the Azure Sentinel service.


2. Choose the workspace where you want to import threat intelligence indicators from the IntSights TAXII Server.


3. Select Data Connectors from the Configuration menu.


4. Select Threat Intelligence – TAXII from the list of the data connectors and click the Open Connector page button.


5. Now enter a Friendly name for the TAXII Server collection. This will be stamped on all the indicators as the Source of the indicator.


6. Now enter the API Root, Collection ID, Username and Password that you obtained from IntSights.


7. Click the Add button.

You should now see a confirmation on the notification dialog that the connection was established successfully. The TAXII Server will now show up in the List of the configured TAXII Servers.

View imported indicators from IntSights TAXII Server in Azure Sentinel

Now that threat intelligence indicators from the IntSights TAXII Server are being imported into Azure Sentinel, you can view them in one of the two below mentioned ways in Azure Sentinel:

a. The ThreatIntelligenceIndicators table in Log analytics

1. Open the Azure portal and navigate to the Azure Sentinel service.


2. Choose the workspace where you imported threat indicators from IntSights using the Threat Intelligence – TAXII Data connector.


3. Select Logs from the General section of Azure Sentinel.


4. The ThreatIntelligenceIndicator table is located under the SecurityInsights group.


5. Use the following query to find indicators from IntSights:

         ThreatIntelligenceIndicator

         | where SourceSystem == “Friendly name of the TAXII Server”

b. Threat Intelligence blade under the Threat Management menu

1. Open the Azure portal and navigate to the Azure Sentinel service.


2. Choose the workspace where you imported threat indicators from IntSights using the Threat Intelligence – TAXII Data connector.


3. Select Threat Intelligence from the Threat management section of Azure Sentinel.


4. You can filter the indicators according to the Source to view the ones that you imported from IntSights. To do so, select the friendly name of the IntSights TAXII Sever you used from the Source pill filter.

Use imported indicators from IntSights TAXII Server in Azure Sentinel

Now that you have imported the indicators into Azure Sentinel, you can use them for matching against log sources. This can be done using the Azure Sentinel out-of-the-box analytics rules by modifying them or by creating new rules from scratch. The set of out-of-the-box analytics rules used to match threat indicators with your event data all have names beginning with, ‘TI map’ under the Analytics section in the Configuration menu. All these rule templates operate similarly with the only difference being what type of threat indicators are used (domain, email, file hash, IP address, or URL) and which event type to match against.

Get insights about your threat intelligence using the new IntSights Workbook

You can use the IntSights Workbook to visualize important information about the threat intelligence you have imported from the IntSights TAXII Server. This workbook provides an interactive dashboard about the threat intelligence you have imported from IntSights. The IntSights Workbook is available on the Azure Sentinel GitHub page. The name of the workbook is “IntsightsIOCWorkbook”.

You can follow the below mentioned steps to get the IntSights Workbook:

1. Open the Azure portal and navigate to the Azure Sentinel service.


2. Choose the workspace where you imported threat indicators from the IntSights TAXII Server.


3. Select Workbooks from the Threat management menu.


4. Navigate to the Templates tab and search for the workbook titled IntSights IOC Workbook.


5. Select the Save button and choose an Azure location to store the workbook.


6. Now select the View saved workbook button to open the workbook for viewing.

Send tags for indicators back to IntSights

You can send a tag to your IntSights account for every IOC that was sent by IntSights and seen in your environment so that you can further analyze the data in the IntSights platform. This can be done simply by deploying a Playbook from the Azure Sentinel GitHub.

Conclusion

With the IntSights and Azure Sentinel integration, you can now gain visibility on indicators of compromise from thousands of sources across the clear, deep, and dark web for advanced detection and hunting threats. You can also receive additional context about the IOCs from the IntSights feed to protect your organization against targeted threats.