Loop Prevention in Exchange Online Demystified

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

We often get questions regarding mail forwarding in Exchange Online. As you already know, Exchange Online is a shared service. We must take care that users cannot take the service down by creating mail loops. It is sometimes a little confusing for our customers that we handle this somewhat differently than in Exchange on-premises organizations. With this blog post we will give you an overview of how we handle possible mail loop scenarios and how this affects your mail flow. If you are the kind of a person that loves digging into the details, this post is for you!

Mail loop insight in the Security & Compliance Center

You may have ended up here because you ran into a possible mail loop scenario. This blog post should help you to get a better understanding about how we handle possible mail loop scenarios, how they can occur and what to do to prevent them. I recommend checking the Fix possible mail loop insight in the Recommended for you area of the Mail flow dashboard in the Security & Compliance Center (protection.office.com) or the converged security portal (security.microsoft.com). It notifies you when a mail loop is detected in your organization.

Read on now if you want to get deeper insights about our loop prevention mechanisms.

Currently stamped and preserved X-headers

It’s important to know that there are some headers which get preserved across Exchange organizational boundaries and most of them will not be removed during transport (neither by Exchange Header Firewall nor using transport rules). These headers are:

• X-MS-Exchange-Inbox-Rules-Loop


• X-MS-Exchange-Transport-Rules-Loop


• X-MS-Gcc-Journal-Report


• X-MS-Exchange-Moderation-Loop


• X-MS-Exchange-Generated-Message-Source


• X-LD-Processed


• X-MS-Exchange-ForwardingLoop


• X-EOPAttributedMessage


• X-EOPTenantAttributedMessage

Please keep in mind that we might change these headers without notice. The purpose of this list is only to provide an overview of different headers we use to prevent loops in the service. Please do not rely on them if you construct business-critical workflows.

To get a better understanding of how loop prevention in Exchange Online works, we will have a look at most of these headers.

X-MS-Exchange-Inbox-Rules-Loop header

First things first: in the cloud, the number of times a message can be redirected, forwarded, or replied to automatically is limited to 1. On-premises Exchange servers are limited to 3 (as documented here).

We set this header for example in the following scenarios:

• User created an inbox rule which forwards the message to another recipient


• User created an inbox rule which redirects the message to another recipient

The value of this header denotes the original recipient of the mail (one which has a mailbox rule to forward to the new recipient in To: header). It looks like this:

X-MS-Exchange-Inbox-Rules-Loop: john.doe@contoso.com

If you run an extended message trace:

Start-HistoricalSearch -ReportType MessageTraceDetail -ReportTitle Inbox-Rules-Loop -MessageID “<1234567890123456789012345678901@AM0PR04MB6531.eurprd04.prod.outlook.com>” -NotifyAddress john.doe@contoso.com -StartDate 05/09/2020 -EndDate 05/10/2020

And you find something like this:

Source: MAILBOXRULE
event_id: THROTTLE
reference: XLoopHeaderCount:1/1

It means that the message got silently dropped because it has reached datacenters X-MS-Exchange-Inbox-Rules-Loop limit which is (as described above) 1.

Mostly important for Exchange Online customers who run an on-premises Exchange organization in a Hybrid configuration: we also check if current recipient mail address is already present within an X-MS-Exchange-Inbox-Rules-Loop header for incoming messages. If that is the case, then we silently drop the message as well (not yet relevant in Exchange Online because the X-MS-Exchange-Inbox-Rules-Loop limit is currently 1 which means we drop the message if any X-MS-Exchange-Inbox-Rules-Loop header exists when the message arrives, regardless of which address the header contains).

X-MS-Exchange-Transport-Rules-Loop header

We set this header in the following scenario:

Mail gets redirected or recipients are added (To, Cc, Bcc) by an Exchange transport rule (ETR)

If the value of this header exceeds its thresholds (in the cloud, the number of times a message can be redirected or forwarded automatically by using a transport rule is limited to 1 – please have a look at Scenario 2 in the Real-life examples – what is the impact on email? how this can happen), we then stop processing, drop the message, log the event, and finally send out an NDR to the original sender like this:

550 5.7.128 TRANSPORT.RULES.RejectMessage; Transport rules loop count exceeded and message rejected

Note: We do not send an NDR to the original sender for any recipient added to Bcc by an Exchange transport rule during the mail flow (we only NDR for To and Cc type recipients).

X-MS-Exchange-Moderation-Loop header

Example scenario when we set this header:

• When a message is forwarded for approval, we stamp this header into the approval message followed by arbitration mailbox SMTP address:

X-MS-Exchange-Moderation-Loop: SPO_Arbitration_fa627f00-12d2-4d68-bd5d-75cd62ead0ee@M365x777241.onmicrosoft.com

• We allow at maximum 1 header. If there are more headers in place, we are going to silently drop the approval message.

You will find the following smtp status logged when running a message trace:

550 5.2.0 Resolver.MT.ModerationLoop; Loop in approval process

X-LD-Processed header

We use this header to track transport processing on a per tenant basis:

• To track potential loops due to mail contact ExternalEmailAddress (TargetAddress) redirection


• If forwarding is configured using the Set-Mailbox -ForwardingSMTPAddress or -ForwardingAddress parameter

If an action like this was detected from our service, we stamp the header followed by the ID of the tenant and a list of strings indicating the list of work that is being tracked.

This may be:

• ExtAddr if we are doing an external redirection by using ExternalEmailAddress


• ExtFwd if we are doing external forwarding by using ForwardingSMTPAddress or ForwardingAddress

If the message is redirected or forwarded to another tenant, we add another X-LD-Processed header containing the tenants ID (we do not replace any existing X-LD-Processed header). If the message comes from an external address and is redirected to another external address, we also stamp the Resent-From header to indicate that Exchange has touched it.

We allow a maximum of 3 loops per tenant for ExternalEmailAddress (TargetAddress) or ForwardingAddress/ForwardingSmtpAddress.

If we exceed the number of forwards, we track the following smtp event (you can find the event by running a message trace). We do not send out an NDR to prevent further loops:

550 5.4.142 RESOLVER.FWD.LoopingTarget; forwarding to a looping external address

We also detect if there is a loop within the directory. You normally should not run into this kind of loop. It can occur, for example, when a mailbox has forwarding configured and ForwardingAddress refers to itself. This job is done while the message is processed. If we detect a loop here, the message will be dropped and we NDR the sender with:

550 5.4.6 RESOLVER.FWD.Loop; there is a forwarding loop configured in the directory

X-MS-Exchange-ForwardingLoop header

This header is added when forwarding happens due to ForwardingSmtpAddress or ForwardingAddress properties set on a mailbox. In the case where the mailbox also has DeliverToMailboxAndForward:$true, when recipient A forwards a message to recipient B, there will be two copies of the message. One to the original recipient A and the other to the forwarded recipient B. The value of the header in the message to the forwarded recipient B will contain <SmtpAddressOfOriginalRecipient>;<TenantGuidOfOriginalRecipient>. The header looks like this:

X-MS-Exchange-ForwardingLoop: JDoe@contoso.com;53bb1ab7-edea-4e35-8c3f-e395807764bf

The purpose of this header is to detect forwarding loops like A forwards to B and B forwards to A.  If B attempts to forward to A, the message will be dropped with the smtp response:

550 5.4.142 RESOLVER.FWD.LoopingTarget; forwarding to a looping external address

The message copy to the original recipient A will also have this header added with the value ForwardingHandled;< TenantGuidOfOriginalRecipient>. It looks like this one:

X-MS-Exchange-ForwardingLoop: ForwardingHandled;53bb1ab7-edea-4e35-8c3f-e395807764bf

The purpose of this header with ForwardingHandled value is to prevent forwarding message multiple times in scenarios like Centralized Mail Transport (aka CMT or CMC), where the message to the original recipient is routed out of the service and then back to the service. In a CMC scenario the message will be forwarded first when the message enters the service. When the message gets routed out and sent back to the service, duplicate forwarding will be prevented by looking at this header in the message. Please have a look at Scenario 5 at the end of this post to get a better understanding of the workflow in CMC scenario.

Note: Customers sometimes make use of the X-MS-Exchange-Inbox-Rules-Loop header to check if a message was forwarded to forwarding address or forwarding SMTP address of the mailbox. If you are doing so, you should now use the new X-MS-Exchange-ForwardingLoop header instead.

X-MS-Exchange-Generated-Message-Source header

This header is used to check for loops in Exchange agent-generated messages. In Exchange Online, we do this while they are in submission and smtp process. We make use of this header for example if an automatic reply via inbox rule is in place. We then stamp the following headers:

X-Auto-Response-Suppress: All
X-MS-Exchange-Inbox-Rules-Loop: john.doe@contoso.com
auto-submitted: auto-generated
X-MS-Exchange-Generated-Message-Source: Mailbox Rules Agent

Let us have a closer look at these X-headers:

• X-Auto-Response-Suppress
  
  
  
  • Specifies whether a client or server application will forego sending automated replies in response to this message. There are different values available. In case of an automatic reply, Exchange sets the value to All.
  
  
  
  


• X-MS-Exchange-Inbox-Rules-Loop
  
  
  
  • Please have a look at 1) X-MS-Exchange-Inbox-Rules-Loop
  
  
  
  


• auto-submitted
  
  
  
  • Defined in RFC 3834. Let me quote from there:
  
  
  
  

“The purpose of the Auto-Submitted header field is to indicate that the message was originated by an automatic process, or an automatic responder, rather than by a human; and to facilitate automatic filtering of messages from signal paths for which automatically generated messages and automatic responses are not desirable.”

• X-MS-Exchange-Generated-Message-Source
  
  
  
  • Short: This X-header is used to prevent mail loops caused by agent-generated messages.
  
  
  • Long: If this header is present, it means that the mail you see is an agent-generated message. We check if it does not exceed our limits. We do these checks while the message is processed in submission and smtp. There are different limits in place. If the message is an inter-tenant one (means no intra-tenant organization header exists), we limit this to 3 agents, and they may be the same (for example three times DLP Policy Agent).
  
  
  
  

You will see multiple agents for example if you have DLP in place and an inbox rule which redirects every message to another mailbox. If your DLP policy matches and you receive a mail notification that gets redirected to another mailbox, you will see something like this:

X-MS-Exchange-Generated-Message-Source: DLP Policy Agent,Mailbox Rules Agent.

If the message is an intra-tenant one, we limit this to a maximum of 2 Exchange agents. Side effect messages are, for example, intra-tenant messages. This kind of messages are generated after a message has been delivered to the mailbox. For example, a message delivered to a mailbox triggers an auto reply or inbox rule that redirects the message to another recipient. In this case, a side effect message is generated. We drop the message if the Exchange agent is the same (for example two times DLP Policy Agent). While in progress, side effect messages are stamped with the following header which is replaced after the message has been delivered.

X-MS-Exchange-Organization-Generated-Message-Source: Mailbox Rules Agent

Some more loop protection insights

We also detect incoming messages that are looping when they pass Exchange Online Protection (EOP). We count every time a message passes through EOP frontdoor and we reject every message that reaches our thresholds. Let me explain this in a little more detail.

To do this, we need some more headers. As this is EOP related work, the headers are named like this:

• X-EOPAttributedMessage


• X-EOPTenantAttributedMessage


• Some more internal loop prevention headers for routing to quarantine and ATP

We increase the X-EOPAttributedMessage header every time the message is processed by EOP frontdoor. It looks like this:

X-EOPAttributedMessage: 1

We also stamp the X-EOPTenantAttributedMessage header with tenant guid and a number which shows how often the message was processed through this tenants EOP. A valid header of a messages that passes EOP for the first time looks like this:

X-EOPTenantAttributedMessage: 543b1ab7-eeea-4a35-8c3f-e396007764bf:0

If the message is re-routed through another tenant (for example, an ETR in Tenant A automatically forwards the message to Tenant B), the X-EOPTenantAttributedMessage header is reset. Anyway, the X-EOPAttributedMessage count is kept and increased. We drop the message if it goes several times through the same tenant (X-EOPTenantAttributedMessage) or when it exceeds a threshold of several more message being routed between different tenants).

If we exceed the number of total hops (which is currently 7 but it but can be changed in the future without being separate announced), you can find the following smtp response logged:

554 5.4.14 Hop count exceeded – possible mail loop ATTR1

If we exceed the hop count within the same tenant (which is currently 3 but can be changed in the future without separate announcement), we then NDR this one out:

554 5.4.14 Hop count exceeded – possible mail loop ATTR34

If you see any of the following smtp responses logged, you then should be open a support case for further investigation. We protocol these if a threshold associated with quarantine or ATP has been reached and a message has been dropped by the service:

454 4.4.15 Hop count exceeded – possible mail loop ATTR39

454 4.4.15 Hop count exceeded – possible mail loop ATTR40

Real-life examples – what is the impact on email?

Here are some examples of how all of this may affect your mail flow:

Scenario 1:

John Doe (Contoso Ltd.) creates an inbox rule to redirect every message to Mike Meyer (TailSpin Toys). Mike in turn has another inbox rule in place to redirect every incoming message to Anna Smith (Fabrikam, Inc.).

Result:

In this case, Exchange stamps the X-MS-Exchange-Inbox-Rules-Loop: John.Doe@contoso.com header after the first redirect is processed. Exchange at TailSpin Toys detects that header (message tracking reference will log XLoopHeaderCount:1/1) and does not redirect the message again. In this case Mike Meyer will not get an NDR. As an administrator you will find this event by running a message trace.

This scenario is one of the most seen and we must be very restrictive because customers can easily build a loop here. Therefore, we restrict this to only 1 redirect/forward by using inbox rules. This limit is hardcoded and cannot be changed. If you make use of Exchange on-premises, it is possible to have up to 3 redirects by inbox rules in place. As an administrator it is your task to protect your users from building loops. 

Scenario 2:

In this scenario we make use of two transport rules. The first one, located at Contoso’s Exchange organization, redirects every message addressed to John.Doe@contoso.com and coming from outside the organization, to Mike Meyer at TailSpin Toys company:

At TailSpin Toys there is also a transport rule in place, to Bcc incoming messages to Anna Smith at Fabrikam:

Result:

In this case the message is stamped at transport within Contoso organization. We see the X-MS-Exchange-Transport-Rules-Loop: 1 header. At TailSpin Toys the message is dropped due to X-MS-Exchange-Transport-Rules-Loop: 1 header. If you run a message trace, you will find the following event logged:

550 5.7.128 TRANSPORT.RULES.RejectMessage; Transport rules loop count exceeded and message rejected

Remember: We do not send out an NDR to the original sender. This is because of the second transport rule which is configured to add an additional recipient as BCC into the message. If the second transport rule is configured to add an CC-Recipient or simply redirects the message instead of an additional recipient as BCC, an NDR will be send out to the original sender.

Scenario 3:

In this scenario we are going to have a look at the moderation loop protection. We have a transport rule (2) that forwards every message, send from outside the organization to shared@fabrikam.com, to Anna Smith for approval (3). Anna is going to holiday and so she has created an inbox rule to redirect every message to the marketing team (4). Unfortunately, the marketing distribution list is also moderated.

Result:

In this case, the moderation message is stamped with the X-MS-Exchange-Moderation-Loop header, followed by the smtp address of the arbitration mailbox:

X-MS-Exchange-Moderation-Loop: SPO_Arbitration_fa627f00-12d2-4d68-bd5d-75cd62ead0ee@M365x777241.onmicrosoft.com

The approval request which is forwarded via inbox rule from Anna’s mailbox to the marketing distribution list is dropped and no NDR is send out to the original sender. If you run a message trace, you will find the following smtp response stamped:

550 5.2.0 Resolver.MT.ModerationLoop; Loop in approval process

Scenario 4:

In this scenario we are going to have a look at the X-LD-Processed header and how it works. Assumed we have the following setup: We have two companies – TailSpin Toys and Fabrikam. Both have marketing departments working together. To make this workflow easier, they decide to create mail users for each other company department (Marketing-Fabrikam and Marketing-TailSpin). Unfortunately, someone added the mail users to the local marketing distribution list – and the loop begins.

Result:

What happens here? Someone sends a mail to one of the distribution lists. Exchange expands the list and starts processing the item. For the first time the mail gets processed, we check on transport if any X-LD-Processed header followed by tenants guid is present. If this is not the case, we stamp a header like this:

X-LD-Processed: 6249f43a-676b-4124-a13a-50205140b751,ExtAddr

We do this also for the other tenant. So one more X-LD-Processed header is added:

X-LD-Processed: 53bb1ab7-edea-4e35-8c3f-e395807764bf,ExtAddr

We keep doing this every time the message is processed and at the end (after the message was processed for the 3^rd time), the header looks like this:

X-LD-Processed: 6249f43a-676b-4124-a13a-50205140b751,ExtAddr,ExtAddr,ExtAddr

X-LD-Processed: 53bb1ab7-edea-4e35-8c3f-e395807764bf,ExtAddr,ExtAddr,ExtAddr

If the message enters transport again, it will be dropped and NDR is send out to the original sender:

554 5.4.14 Hop count exceeded – possible mail loop ATTR1 [HE1EUR04FT048.eop-eur04.prod.protection.outlook.com]

And so, the loop ends.

Scenario 5:

In this scenario we are going to have a look at the X-MS-Exchange-ForwardingLoop header and how it works. Assumed we have the following setup: Fabrikam has a Hybrid configuration and have also enabled Centralized Mail Transport (CMT; also known as CMC, RouteAllMessagesViaOnPremises enabled on the outbound connector). MX points to the Exchange Online service to make use of our malware and spam protection features. They need to route all outgoing messages through Exchange on-premises because of their mail signature and DLP solution which has not been migrated to cloud yet. Anna Smith of Fabrikam has set ForwardingSmtpAddress to John.Doe@contoso.com as well as DeliverToMailboxAndForward set to $true.

Result:

A message addressed to Anna.Smith@fabrikam.com enters the service (1). We notice that ForwardingSmtpAddress is set to John.Doe@contoso.com as well as DeliverToMailboxAndForward is set to $true. Now message bifurcation kicks in and creates another copy of the message. The message which goes to John.Doe@contoso.com gets stamped (2a) with following header:

X-MS-Exchange-ForwardingLoop: Anna.Smith@fabrikam.com;53bb1ab7-edea-4e35-8c3f-e395807764bf

And is routed through Exchange on-premises to John.Doe@contoso.com. Unfortunately, there is also ForwardingSmtpAddress set on John Doe’s mailbox which points to Anna.Smith@fabrikam.com. The message is now routed to the Exchange Online service again (3a). Here we detect that X-MS-Exchange-ForwardingLoop has already been set with that recipient and tenant guid. Result of this is that we drop the message with smtp response:

550 5.4.142 RESOLVER.FWD.LoopingTarget; forwarding to a looping external address

The original message copy to Anna Smith’s mailbox is also on its way. This message also makes an extra round through Exchange on-premises due to requirement imposed by having CMC enabled and it’s coming from Internet (i.e. not having passed through on-premises yet). Before being sent to on-premises, however, this copy was stamped with a header (2b). It looks a little different to the header mentioned before:

X-MS-Exchange-ForwardingLoop: ForwardingHandled;53bb1ab7-edea-4e35-8c3f-e395807764bf

As you can see, it contains ForwardingHandled instead of the recipient’s e-mail address. We need to do this in order to remember that Forwarding was already applied in cases when the message then has a routing requirement out of the EXO service such as Centralized Mail Transport. When on-premises sends it back to Exchange Online (3b), we now figure out that the X-MS-Exchange-ForwardingLoop header has already been stamped with ForwardingHandled flag. Result of this is that we don’t forward the message again (no RESOLVER REDIRECT event in Message Tracking Log/Message Trace Details Report). The message is finally delivered to Anna Smith’s mailbox (4b).

What if I need to do multiple forwards or redirects for handling business processes?

Glad you asked! Basically, there are 3 ways to go:

• Keep these mailboxes on-premises (which is not a real option if you want to use the awesome features available in Exchange Online). Remember: Some limits are different when using Exchange on-premises.


• Configure redirect via Set-Mailbox -ForwardingSMTPAddress or -ForwardingAddress parameter. As described above, we make use of the X-LD-Processed header in this scenario which allows up to 3 redirects per tenant.


• Make use of Microsoft Power Automate. With Power Automate you can create automated workflows between your favorite apps and services in a centralized way. This includes many email actions when using the Microsoft 365 Outlook Connector. I encourage you to check out the documentation, give it a try and see the power of this Microsoft Power Platform application.

I hope all these insights help you to get a better overview and understanding of how we handle possible loop situations in Exchange Online. It is sometimes a little confusing and may look complicated, but keeping the service running for our customers is a must.

Special thanks to all contributors: Especially to Stan Aleksiev for taking care of the technical review, Arindam Thokder, Dan Li, Guru Prasad, Arnold Kermer and Dmitry Starostin!

Lukas Sassl

Log Analytics pinned parts now works with Azure Dashboard filters

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Intro:

As we continue to improve our Log Analytics pinned parts experience to Azure Dashboards, we are happy to announce integration with dashboard filters.

Integration with Dashboard filters:

Log Analytics pinned parts are now integrated with dashboard filters – you can now add a filter to your
dashboard and it will apply to pinned Log Analytics parts:

Use the new filtering experience to achieve more with Azure dashboards.

Feedback:

We appreciate your feedback! comment on this blog post and let us know what you think of the this
feature.
You may also use our in app feedback feature to provide us with additional feedbacks:

Log Analytics UI – New experience for Custom Logs

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Intro

We continue to improve our experiences!
The Custom Logs and custom fields screens get a new, dedicated experience in your log Analytics workspace.  

The new Custom Logs Blade:

Reach your custom logs blade from the left hand navigation bar in your Log Analytics workspace:

The new experience was updated with a cleaner look and feel, for custom logs:

And custom fields:

The new experience also allows filtering of the custom logs or custom fields for easier management:

Feedback

We value your feedback, please let us know what you think by commenting on this blog post or by clicking the ‘feedback’ button right in Log Analytics:

Understanding Microsoft Azure Virtual Machine sizes

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Having an on-premises infrastructure background, I’m used to scoping hardware by defining the specifications (CPU, memory etc) we’ll need to run the applications and expected concurrent users and allowing for some growth. Then we’d often buy a box that would give us room to upgrade further, so for example we’re not putting the maximum amount of RAM in that server today. But when it comes to creating a virtual machine in Microsoft Azure, you’re now faced with unfamiliar choices – Burstable, D series, F series etc. and a mix of vCPU, RAM and temporary storage combinations. How do you know which 8 vCPU 32GB size to choose?

T-shirt sizes!

A great way to think of the different combinations of specifications is to relate it to clothing. If I’m buying a t-shirt, I can choose from small, medium, large, extra large etc. Each size has specifications for the body length, sleeve circumference, neck circumference etc. I also have to decide how I like my t-shirts to fit – do I want a tighter, slimmer fit or a baggier, relaxed fit? A small size in a slim but range will be different to a small size in a relaxed cut. And what style do I want – am I going for short sleeves, no sleeves, long sleeves, round neck, or v neck? There are a lot of decisions we make when we’re buying clothing, but these choices are familiar.

Microsoft uses the terms category, series and instance when talking about virtual machine sizes. Lets start with our VM “t-shirt” style!

Category

You’ll find the high level categories mentioned in some of the Azure VM documentation, include the pricing information. This is a great place to start to narrow down which machines would be the most suited to the workloads you want to run. Pick your style!
General purpose: Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers.
Compute optimized: High CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes and application servers.
Memory optimized: High memory-to-core ratio. Great for relational database servers, medium to large caches and in-memory analytics.
Storage optimized: High disk throughput and IO. Ideal for Big Data SQL, and NoSQL databases.
GPU: Specialised virtual machines targeted for heavy graphic rendering and video editing available with single or multiple GPUs.
High performance compute: Our fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA).

Series

Next, you’ll choose your t-shirt fit, by examining the different series of virtual machines. A series is a group of virtual machine sizes based on the same host hardware configuration. For compute optimized, that’s CPU focussed. For storage optimized, that might be local SSD disks or directly mapped local NVMe storage.

In the Compute optimised category for example, lets compare two different series:
Fsv2 series: 2GB RAM and 8GB local temporary storage per vCPU, hyperthreaded and based on the Intel Xeon Platinum 8272CL (second generation Intel Xeon Scalable processors or the Intel Xeon Platinum 8168 (Skylake) processor.

F series: 2GB RAM and 16GB local temporary storage per CPU core, based on the Intel Xeon Platinum 8272CL (second generation Intel Xeon Scalable processors, Intel® Xeon® 8171M 2.1GHz (Skylake), Intel® Xeon® E5-2673 v4 2.3 GHz (Broadwell) or the Intel® Xeon® E5-2673 v3 2.4 GHz (Haswell) processor.

Instance

Now we’re down to the specific t-shirt size for your virtual machine, it’s own combination of CPU, RAM and temporary storage. Looking again at computer optimized machines, we’ll see the Fsv2 series offers vCPUs in combinations like:

While the F series offers physical CPU cores in combinations like:

Note: Temporary storage is a non-persistent disk that disappears & is recreated new if the VM is shut down, resized, moved to a different physical host or if the host is updated or upgraded. It’s the default location for the pagefile.sys for Windows and can also be used for SQL’s TempDB. You need to provision additional storage for your applications and data, which is not included in the instance sizes.

Where to find sizing information

As sizing is a consideration when estimating or creating a virtual machine, you can find a direct link to detailed sizing information from the Azure Pricing Calculator when you add a virtual machine to your estimate. After adding a virtual machine, click the i symbol then choose Pricing details:

Azure Pricing Calculator link to VM sizing information

And from the Azure portal when you create a new virtual machine, click the i symbol next to Size, then choose Learn more about Virtual Machine sizes:
Azure portal VM size information

B-series burstable virtual machines

The B-series VMs are unique in that they will build up credits when they are operating under their baseline CPU performance, but they can use more than that baseline when your application needs it, up to the maximum provided by the instance size you have selected. For example, the Standard_B2ms instance has 2 vCPUs and a baseline of 60% CPU performance of the VM. When the VM is operating at less than 60% of the CPU performance, you’ll accumulate credits. When needed, the Standard_B2ms can burst up to 200% max CPU performance. This is great for applications that have regular, short periods of high demand and long periods of low or no demand, like outside of office hours.

In the Azure Pricing Calculator, there’s a switch you can toggle to not show B-series VMs in the size selector. For more information on burstable VMs, maximum credits and how they are applied, visit B-series burstable virtual machines sizes.

Restricting certain VM sizes

Azure Policy lets you control which VM sizes are allowed to de deployed in your environment, and by omission, which sizes are not allowed. The instance sizes are referred to as SKUs (stock keeping units) and this Deny policy will stop VMs being created with or resized to any instance that is not listed in your policy. This is an effective way of putting a cost control measure in place to ensure that the more expensive sizes are not deployed without your knowledge. For more information see Azure Policy built-in definitions for Azure Virtual Machines and the json policy definition on GitHub. 

Getting your sizing right

If you are still unsure, the Azure Portal will recommend sizes related to the Image you have selected for your VM. When an operating system image is added to the gallery, the publisher can recommend a list of instance sizes that are appropriate for that image:

Azure portal showing recommended sizes for the selected image

Note: Note all instance sizes are available in all Azure regions due to capacity of the data centers and demand. As you need to choose a region first when you create a VM in the Azure portal, you’ll received a warning in red if the selected size is not available in that region. Instance sizes can also be restricted depending on your subscription type. For example, an Azure free account is limited to 750 hours of Azure B1S virtual machines for the first 12 months. Also, as per the previous section, you’ll receive an error if an administrator has prevented the creation of (and resizing to) certain instance sizes.

List your VMs and their sizes

You can use the Azure Resource Graph Explorer to return a list of all your VMs and their current sizes, with the following KQL query:

| where type =~ 'Microsoft.Compute/virtualMachines'
| project vmName = name, vmSize=tostring(properties.hardwareProfile.vmSize), vmId = id

Resizing

If you find that your virtual machine is not performing as well as you need it to, you may have undersized it. Azure Advisor will also notify you if your VM is consistently under utilised and could be down sized, saving you money. Fortunately, resizing a VM is a simple process using the Azure portal or PowerShell, but it does require your VM to be shut down and restarted. Also, if your VM is part of an availability set and the new size is not available in it’s current physical host hardware cluster, all of the VMs in that availability set need to be deallocated and moved, which may require updating the size of the other VMs too. For detailed information on resizing a virtual machine, visit Resize a Windows VM in Azure. 

Learn more:

Docs – Sizes for virtual machines in Azure (including a great video)

MS Learn – Introduction to Azure virtual machines

How to Quick Start with Defender for IoT Sensor onboarding and integration into Azure Sentinel

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Azure Defender for IoT is a unified security solution for identifying IoT/OT devices, vulnerabilities, and threats. It enables organizations to secure entire IoT/OT environments, whether there is a need to protect existing IoT/OT devices or build security into new IoT innovations.

Azure Defender for IoT offers agentless network monitoring that can be deployed on physical hardware or virtualized environment and a lightweight micro agent that supports standard IoT operating systems. OT (Operational Technology) is used to monitor Industrial equipment rather than traditional Network IT resources.

Azure Sentinel can be used to integrate with Defender for Security Orchestration, Automation, and Response (SOAR) capabilities enables automated response and prevention using built-in OT-optimized playbooks.

This Blogpost presents two topics to support enterprises and enable a quick start with IoT/OT:

• Onboard an agentless Defender for IoT sensor for PoC/Evaluation purpose.


• Integration of Defender for IoT with Azure Sentinel for unified security management across IoT/OT landscape.

Prerequisites and Requirements

This capture describes the requirements to set up the environment.

• Hardware appliance for the sensor.

The supported hardware for Defender IoT is listed here: Identify required appliances – Azure Defender for IoT | Microsoft Docs

• A network switch that supports traffic monitoring via SPAN port.


• Create or use an existing Azure IoT Hub service. IoT Hub is required to manage IoT devices and security.


• An existing Azure Sentinel deployment for unified security management experience for Defender for IoT alerts.

Install the Defender for IoT Sensor

The installation takes a while and requires several reboots during the installation.

Before you can start the installation, there is a need to download the installation software. The ISO for the installation can be found in Azure Portal > Azure Defender for IoT > Set up a sensor > Purchase an appliance and install software > Download.

For my lab environment, I decided to use a Vmware ESXI server. I created a guest VM with 4 CPU cores, 8 GB of RAM, 128 GB of hard drive, and 2 virtual network cards for the sensor. One virtual card will be later used for the management interface, and the second one for the SPAN port. I prepared the environment for my lab as follow:

For installing the sensor, I attached the downloaded ISO to the sensor guest VM to kick off the installation.

For the initial configuration, select a language.

Select SENSOR-RELEASE-version Office.

Configure the architecture and the network properties.

Use eth0 for the management network (interface) and eth1 for the input interface (SPAN port) and click “y” to accept the configuration.

After few minutes, CyberX and support credentials appear. Copy the passwords for later usage.

• Support: The administrative user for user management.


• CyberX: The equivalent of root for accessing the appliance.

Select Enter to continue.

Once the installation is finished, you can access the management console via the configured IP address during the installation.

                https://ipaddress

Onboard the agentless Sensor in Event Hub

Once the sensor is installed, now it’s time to prepare the sensor as a cloud-connected sensor. In this mode, the sensor would send the alerts to Event Hub to share them with Azure services such as Azure Sentinel.

For the next step, there a need for an activation file. The Activation files contain the instructions for the management mode of the sensor.

To get the activation file, perform the following steps.

From the Azure Portal, navigate to Defender for IoT > Start discovering your network / Onboard sensor.

Define a name for the sensor, choose the subscription, select On the cloud, select an IoT Hub or create one, use a Display name and click to Register.

Now the Activation file is generated and can be downloaded for the next step. Download the file and save it for the next step to activate the sensor in cloud-connected mode.

Activate the agentless Sensor

The following steps are required to activate the sensor and to perform the initial setup.

Log on to the management console from your browser and the CyberX credential, which was pre-defined, including password during the installation.

After sign in from the Activation page, upload the Activation File, which was saved in preview steps, approve the Terms and Conditions and click Activate.

After activation, I would recommend some best practices to follow:

• Create a new Admin account for management and only use the CyberX and support account if there is a need for it.


• Change the sensor’s name and, if required, the network settings in the network configuration settings.

Validate the Sensor

After logging in to the management console, the sensor can be validated.

I see the SPAN input is functional, and data is streamed from the mirror port.

The sensor also discovered the asset as well as built a network map based on the discovery.

Integrate with Azure Sentinel

As the sensor is operated in a cloud-connected mode, the integration into Azure Sentinel is a one-click experience.

To enable the data connector in Azure Sentinel, open the Azure Portal and navigate to Azure Sentinel > Data connectors and search for the Azure Defender for IoT connector, then click to Open connector page.

And click to connect your Subscription to stream IoT Hub alerts into Azure Sentinel.

In the Next Steps selection, you can enable the Create incidents based on Azure Security Center for IoT alerts analytics rule to create incidents that Azure Sentinel can manage.

Additionally, use the Azure Defender for IoT Alerts workbook to gain insights into your IoT data workloads from Azure IoT Hub managed deployments, monitor alerts across all your IoT Hub deployments, and detect devices at risk act upon potential threats.

With the enabled data connector, you can manage the Defender for IoT incidents in Azure Sentinel. Please check the SecurtityAlert table for all the alert data from Defender for IoT. 

SecurityAlert | where ProductName == “Azure Security Center for IoT”

| sort by TimeGenerated

Or from the Azure Sentinel Incident dashboard.

Summary

In this blog post, I covered the deployment of an agentless Defender for IoT sensors and the integration with Azure Sentinel to manage the security incidents.

Stay tuned for other IoT-related content in this channel.

Additional Resources

Azure Defender for IoT Landing Page

https://azure.microsoft.com/en-us/services/azure-defender-for-iot/

Agentless IoT/OT Security with Azure Defender for IoT

https://www.youtube.com/watch?v=8spIfxewaeM&feature=youtu.be

Thank you for

Additionally, many thanks to Paul Roberts and Clive Watson for brainstorming and ideas for the content.

Set Your Alert State Using Azure Automation

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Hello blog readers 

One of recurring questions during my customer engagements on Azure Monitor is: how do I set alert state to either Acknowledged or Closed with no manual intervention?

This question is broader and deeper than it appears. In fact, linked to the pure and simple alert state there are often ITSM processes coming along. State is just an alert property that can have only 1 of the 3 following values at a given time: New, Acknowledged or Closed. Should you want to read more about Azure Monitor alerts (including their states) you can find more information in the official Microsoft documentation at Overview of alerting and notification monitoring in Azure – Azure Monitor | Microsoft Docs.

Hence, when it comes to the state, we also need to consider other actors. In a simple scenario, where have notifications and no ITSM processes, we can automate alert state management using Azure Automation to fire a runbook that sets the alert state on schedules. Differently, on mature customers or high integrated IT environments, where alerts are part of the incident management process(es), we must consider that alert states have to be managed in line with the ITSM integration. The below diagram quickly describes the scenario for alerts lifecycle when the ITSM integration is in place:

Azure Monitor <–> ITSM integration flow

So, provided that you have evaluated the best scenario according to the company’s business needs, the idea shared here is very easy and works very well especially with metrics-based alerts where you have a stateful alert approach.

With log-search based alerts, the situation can become a bit more complex since these alerts are stateless.

Looking at the alerts from Azure Monitor – Alerts blade,

Azure Monitor Alert Dashboard

you may have noticed that among all the columns, we have one called Monitor condition, whose value is sometime set to Fired or Resolved, and one called Signal type.

Let us start with the Signal type one. This one stands for the repository (and hence the type of data we are going to use for the alert: Metrics or Logs) where the data is stored. It is important to understand that because the type of data is what drives the value in the Monitor condition column. This column is showing the status of the object/aspects we created the alert for.

But why it sometime shows as resolved and sometimes not? The answer is exactly in the value reported by the Signal type column. When Signal type is Metrics or Health, it means that we are using data whose certainty is guaranteed 100%. In other words, that type of data will always be produced, collected and stored in Azure, so we can check whether an issue has been resolved or not and set the Monitor condition property value accordingly. This certainty makes the alerts stateful. For more info you can check the Understand how metric alerts work in Azure Monitor documentation at https://docs.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-metric-overview

Differently, when it is Log there is no assurance that we either collected or received data. Think about an on-prem environment in which we have several dependencies as part of the trip to Azure Log Analytics. Think about what happen when we lose Internet connectivity or the monitoring agent just stops, or the server is powered off. How can we make sure the issue is resolved if we have no data confirming it? This uncertainty makes the log-based alerts stateless. Should you need more info, you can refer to the Log alerts in Azure Monitor documentation, specifically looking at the State and resolving alerts paragraph.

With all that said we now have a better idea of what to do to set our alert state in both scenarios (Metrics/Health and logs).

Since we proved so far that using Metrics or Health as signal type we always have the correct and up-to-date condition, we can just look at that the MonitorCondition property value and set the alert state to Closed. In that case the simple automation runbook I am suggesting below can help:

<#
.SYNOPSIS 
    This sample automation runbook is designed to set the metric or health based alerts to Closed.
    

.DESCRIPTION
    This sample automation runbook is designed to set the metric or health based alerts to Closed. It looks for all the alerts in the provided time range and for each,
    it will check the value of the MonitorCondition property. Should it be equal to Resolved, we set the state property to Closed.
    This runbook requires the Az.AlertsManagement PowerShell module which can be found at https://docs.microsoft.com/en-us/powershell/module/az.alertsmanagement/?view=azps-5.6.0

    NOTE: TimeRange parameter only accepts the value reported in the ValidateSet. This is in line with the underlying API requirements that is 
    documented at https://docs.microsoft.com/en-us/rest/api/monitor/alertsmanagement/alerts/getall#timerange


.PARAMETER TimeRange
    Required. The TimeRange on which we query the alerts.


.EXAMPLE
    .Close-ResolvedAlerts.ps1 -TimeRange 1d

.NOTES
    AUTHOR:   Bruno Gabrielli
    VERSION:  1.0
    LASTEDIT: Dec 08th, 2020
#>

#Parameters
param(
    [ValidateSet('1h', '1d', '7d', '30d')]
    [string] $TimeRange = '1d'
)

#Inizialiting connection to the AutomationAccount
[String]$connectionName = "AzureRunAsConnection"
try
{
    #Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName

    #"Logging in to Azure..."
    $nullOut = (Add-AzAccount `
     -ServicePrincipal `
     -TenantId $servicePrincipalConnection.TenantId `
     -ApplicationId $servicePrincipalConnection.ApplicationId `
     -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint `
     -WarningAction:Ignore)
        
    #"Setting context to a specific subscription"  
    $nullOut = (Set-AzContext -SubscriptionId $servicePrincipalConnection.SubscriptionId -WarningAction:Ignore)

    $inactiveAlerts = (Get-AzAlert -MonitorCondition Resolved -State New -TimeRange $TimeRange)

    if($inactiveAlerts)
    {
        foreach($alert in $inactiveAlerts)
        {
            Write-Output "Setting state to 'Closed' for alert '$($alert.Name)' which had the monitor condition set to '$($alert.MonitorCondition)' and the state set to '$($alert.State)'"
            Update-AzAlertState -AlertId $alert.Id -State Closed
        }
    }
    else
    {
        Write-Output "No inactive (Resolved) alerts in the specified '$($TimeRange)' period."
    }
}
catch
{
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    }
    else
    {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
} 

As opposite to Metrics or Health based alerts, the Log based alerts need to be managed differently. Here we must look first for the MonitorService property value, making sure that it is equal to “Log Analytics”. After that we need to make some assumptions based on the LastModified property value. Based on the log-based alerts nature, we might assume that if an alert has not been changed later than the TimeRange parameter value we provided, we could close it. We will get a new one soon if the corresponding issue has not been resolved in the meantime. Here below you can find another sample runbook for that purpose:

<#
.SYNOPSIS 
    This sample automation runbook is designed to set the Log Analytics based alerts to Closed.
    

.DESCRIPTION
    This sample automation runbook is designed to set the Log Analytics based alerts to Closed. It looks for all the alerts in the provided time range and for each,
    it will check the value of the MonitorService property. Should it be equal to Log Analytics and last modified later than TimeRange, we set the state property to Closed.
    This runbook requires the Az.AlertsManagement PowerShell module which can be found at https://docs.microsoft.com/en-us/powershell/module/az.alertsmanagement/?view=azps-5.6.0

    NOTE: TimeRange parameter only accepts the value reported in the ValidateSet. This is inline with the underlying API requirements that is
    documented at https://docs.microsoft.com/en-us/rest/api/monitor/alertsmanagement/alerts/getall#timerange


.PARAMETER TimeRange
    Required. The TimeRange on which we query the alerts.


.EXAMPLE
    .Close-ResolvedAlerts.ps1 -TimeRange 1d

.NOTES
    AUTHOR:   Bruno Gabrielli
    VERSION:  1.0
    LASTEDIT: Jan 21st, 2021
#>

#Parameters
param(
    [ValidateSet('1h', '1d', '7d', '30d')]
    [string] $TimeRange = '1d'
)

#Inizialiting connection to the AutomationAccount
[String]$connectionName = "AzureRunAsConnection"
try
{
    #Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName

    #"Logging in to Azure..."
    $nullOut = (Add-AzAccount `
     -ServicePrincipal `
     -TenantId $servicePrincipalConnection.TenantId `
     -ApplicationId $servicePrincipalConnection.ApplicationId `
     -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint `
     -WarningAction:Ignore)
        
    #"Setting context to a specific subscription"  
    $nullOut = (Set-AzContext -SubscriptionId $servicePrincipalConnection.SubscriptionId -WarningAction:Ignore)

    $inactiveAlerts = (Get-AzAlert -MonitorService 'Log Analytics' -State New -TimeRange $TimeRange)

    if($inactiveAlerts)
    {
        foreach($alert in $inactiveAlerts)
        {
            if($alert.LastModified -le ((Get-date).add(-$TimeRange)))
            {
                Write-Output "Setting state to 'Closed' for alert '$($alert.Name)' which had the monitor service equal to $($alert.MonitorService), monitor condition set to '$($alert.MonitorCondition)' and the state set to '$($alert.State)'"
                Update-AzAlertState -AlertId $alert.Id -State Closed
            }
        }
    }
    else
    {
        Write-Output "No inactive (Resolved) alerts in the specified '$($TimeRange)' period."
    }
}
catch
{
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    }
    else
    {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
} 

Both sample runbook codes requires the Az.AlertsManagement PowerShell module to be imported into your Automation Account.

NOTE: As you can see from the script’ comments, TimeRange parameter only accepts the values reported in the ValidateSet. This is in line with the underlying API requirements that is documented at https://docs.microsoft.com/en-us/rest/api/monitor/alertsmanagement/alerts/getall#timerange

With all the ingredients and knowledge, you just have to import the 2 scripts as new runbooks:

Azure Automation Runbooks

and schedule them to run on your preferred interval which can be different from the value you used as TimeRange parameter:

Azure Automation Schedules

Thanks for reading this one till the end,

Bruno.

Disclaimer

The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Meet a Recent Microsoft Learn Student Ambassador Graduate: Ayush Chauhan

by Contributed | May 2, 2021 | Technology

This article is contributed. See the original author and article here.

This is the next installment of our blog series highlighting Microsoft Learn Student Ambassadors who achieved the Gold milestone and have recently graduated from university. Each blog in the series features a different student and highlights their accomplishments, their experience with the Student Ambassadors community, and what they’re up to now. 

Today we meet Ayush Chauhan who is from India and graduated in December from JECRC University located in the city of Jaipur in Rajasthan in India. All the students interviewed so far have been very forthcoming in sharing their history and their experience, but Ayush kicked off the interview with a declarative “I have lots of things to say”, which was terrific! 

Responses have been edited for clarity and length.  

When you became a Student Ambassador in 2017, did you have specific goals you wanted to reach, such as attaining a particular skill or quality? What were they? Did you achieve them? 

I applied on the Student Ambassador website. I was just obsessed with Microsoft from the Windows Lumia age, and I didn’t know what the future was holding for me. My goal was to be able to write code or build software that will help people, or that will impact the developer and different communities. When I was creating a video to submit on the application, it was the first time I got my hands on Node.js & Bot framework. I learned it using Microsoft Docs, and I have never stopped learning since then. So yes, I was able to achieve my goal. 

How has being in the Student Ambassador community impacted you in general besides helping you develop additional tech skills?   

I landed my first internship in the second year just because of the bot I developed for my Student Ambassador application. 

Being in this program, I got to learn from experts, and it has impacted my life because I was winning competitions, around 10 hackathons. It gave me huge confidence too that I can build anything I can think of and go on any stage to represent it. Microsoft has impacted me a lot in three years. It has accelerated my learning and ability to build anything I can imagine. 

What were the accomplishments that you’re the proudest of? And please give us details.

I won the 2018 India Capgemini Tech Challenge.  I was in my third year at university, and over 3,500 working professionals participated in the Azure category. We had to build a chatbot, so I built a SAAS to help book writers to format or digitalize their writings without needing to wait for a person to write. It was the first time I realized that I can do anything, and age and appearance don’t matter. The only thing that matters is hard work and practice. 

I built a dataset of 100 women’s colleges to help with the diversity and inclusion in our events. It created an opportunity to invite 5000+ STEM students to participate in global events and feel included in the tech community. 

I was proud of the projects that I built in hackathons whether I won them or not. They involved everything from helping elder people with IoT home automation, to a chatbot for newborn children’s parents that can resolve their queries, and much more. 

You graduated a couple few months ago. So what have you been up to since graduation? 

After graduation I joined the School of Accelerated Learning, a startup [editor’s note: India’s first-ever hybrid coding bootcamp for millennials looking to build tech-focused careers]. I have been working to build quality and relevant education for the tech world. So, it is exactly what I believe in. It’s exactly how this program has empowered me to do. We don’t believe in theoretical curriculums or traditional classrooms. We believe in getting everyone ready for the future despite them being from diverse backgrounds. We tell them how to build industry-driven products by themselves. We explain them concepts, we try to build their mindset. We do activities that help them grow their innovation and built a tech-enabled environment that nurtures their growth mindset. 

And I was working in the open-source community. Every time you go on GitHub and see a repository with a “deploy to Azure button” – I made that button, I redesigned it.  

If you could redo things, is there anything you would have done differently while you were a Student Ambassador? Or would you have done things the same? 

I don’t think I would try to redo something because what’s happened, happened. My failures made me what I am today.     

If you were to describe the Student Ambassadors community to a student who is interested in joining, what would you say to them? 

I’ll say “Hey, do you want to make some cool like-minded friends from all over the world? Do you want to gain knowledge and experience the future of productivity with Microsoft? Do want to have the benefits of Visual Studio Enterprise subscriptions?. Also do you want to learn from industry experts and also get a Microsoft certification? Well, this program has covered all of these benefits in a single package, so you won’t stop learning because of less resources or no exposure.” 

What advice would you give to new Student Ambassadors? 

Always have the audacity to curiously ask questions. There is always a solution of how you can solve a hard-coded error. For that you need to avoid a know-it-all mindset. Don’t just react on knowledge you’ve heard or seen.  Go a step ahead, try to learn it all, implement it all. Whatever you want to build, whatever you see, whatever you want to know or add to your skill set, you should just go and learn it all. Learning is something that doesn’t expire with age. 

What is your motto in life, your guiding principle?  

I go with the flow always. I never say no to any opportunity even if I know I’ll be failing. I wake up every day knowing there’s something for me to learn.  Nothing worth having comes easy. There’s so much love and energy to get up and run again, even after you fall if you love what you do. Also, I take care of burnouts.  It’s surprising how something you love so much can hurt you a lot. I take breaks to recover, so I play games and listen to music in that time. 

Good luck to you in the future, Ayush! 

Protect API's using OAuth 2.0 in APIM

by Contributed | May 2, 2021 | Technology

This article is contributed. See the original author and article here.

The API Management is a proxy to the backend APIs, it’s a good practice to implement security mechanism to provide an extra layer of security to avoid unauthorized access to APIs.

Configuring OAuth 2.0 Server in APIM merely enables the Developer Portal’s test console as APIM’s client to acquire a token from Azure Active Directory. In the real world, customer will have a different client app that will need to be configured in AAD to get a valid OAuth token that APIM can validate.

Prerequisites

To follow the steps in this article, you must have:

• Azure subscription


• Azure API Management


• An Azure AD tenant

API Management supports other mechanisms for securing access to APIs, including the following examples:

• Subscription keys
  End-users who need to consume the APIs must include a valid subscription key in HTTP requests when they make calls to those APIs.

• Client Certificate
  The second option is to use Client Certificates. In API Management you can configure to send the client certificates while making the API calls and validate incoming certificate and check certificate properties against desired values using policy expressions.

• Restrict caller Ips
  The third option is Restrict caller Ips – It (allows/denies) calls from specific IP addresses and/or address ranges which is applied in the <ip-filter>Policy.
  
  


• Securing the Back End API using OAuth2.0
  Another option is using OAuth 2.0, Users/services will acquire an access token from an authorization server via different grant methods and send the token in the authorization header. In the inbound policy the token can be validated.  

Azure AD OAUTH2.0 authorization in APIM

OAUTH 2.0 is the open standard for access delegation which provides client a secure delegated access to the resources on behalf of the resource owner.

Note: In the real world, you will have a different client app that will need to be configured in AAD to get a valid OAuth token that APIM can validate. 

The below diagram depicts different client applications like Web application/SPA, Mobile App and a server process that may need to obtain a token in Non-Interactive mode. So you must create a different App Registration for the respective client application and use them to obtain the token.

In this Diagram we can see the OAUTH flow with API Management in which:

1. The Developer Portal requests a token from Azure AD using app registration client id and client secret.


2. In the second step, the user is challenged to prove their identity by supplying User Credentials.


3. After successful validation, Azure AD issues the access/refresh token.


4. User makes an API call with the authorization header and the token gets validated by using validate-jwt policy in APIM by Azure AD.


5. Based on the validation result, the user will receive the response in the developer portal.

Different OAuth Grant Types​:

High-level steps required to configure OAUTH

To configure Oauth2 with APIM the following needs to be created:

• Register an application (backend-app) in Azure AD to represent the protected API resource.​


• Register another application (client-app) in Azure AD which represent a client that wants to access the protected API resource.​


• In Azure AD, grant permissions to client(client-app) to access the protected resource (backend-app).​


• Configure the Developer Console to call the API using OAuth 2.0 user authorization.​


• Add the validate-jwt policy to validate the OAuth token for every incoming request.​​

Register an application (backend-app) in Azure AD to represent the API.​

To protect an API with Azure AD, first register an application in Azure AD that represents the API. The following steps use the Azure portal to register the application.

Search for Azure Active Directory and select App registrations under Azure Portal to register an application:

1. Select New registration.
   
   

    

   
   


2. In the Name section, enter a meaningful application name that will be displayed to users of the app.


3. In the Supported account types section, select an option that suits your scenario.


4. Leave the Redirect URI section empty.


5. Select Register to create the application.
   


6. On the app Overview page, find the Application (client) ID value and record it for later.

1. Select Expose an API and set the Application ID URI with the default value. Record this value for later.


2. Select the Add a scope button to display the Add a scope page. Then create a new scope that’s supported by the API (for example, Files.Read).


3. Select the Add scope button to create the scope. Repeat this step to add all scopes supported by your API.


4. When the scopes are created, make a note of them for use in a subsequent step.
   
   

    

   
   

Register another application (client-app) in Azure AD to represent a client application that needs to call the API.​

Every client application that calls the API needs to be registered as an application in Azure AD. In this example, the client application is the Developer Console in the API Management developer portal.

To register another application in Azure AD to represent the Developer Console:

1. Follow the steps 1 – 6. mentioned in the previous section for registering backend app.

1. Once the App registered, On the app Overview page, find the Application (client) ID value and record it for later.


2. Create a client secret for this application to use in a subsequent step.




1. From the list of pages for your client app, select Certificates & secrets, and select New client secret.


2. Under Add a client secret, provide a Description. Choose when the key should expire and select Add. When the secret is created, note the key value for use in a subsequent step.

Authorization Code​:

In Authorization code grant type, User is challenged to prove their identity providing user credentials.
Upon successful authorization, the token end point is used to obtain an access token.

The obtained token is sent to the resource server and gets validated before sending the secured data to the client application.

Enable OAuth 2.0 in the Developer Console for Authorization Code Grant type

At this point, we have created the applications in Azure AD, and granted proper permissions to allow the client-app to call the backend-app.

In this demo, the Developer Console is the client-app and has a walk through on how to enable OAuth 2.0 user authorization in the Developer Console.
Steps mentioned below:

1. In Azure portal, browse to your API Management instance and Select OAuth 2.0 > Add.


2. Provide a Display name and Description.


3. For the Client registration page URL, enter a placeholder value, such as http://localhost.


4. For Authorization grant types, select Authorization code.
   
   
   

    

   
   


5. Specify the Authorization endpoint URL and Token endpoint URL. These values can be retrieved from the Endpoints page in your Azure AD tenant.

        Browse to the App registrations page again and select Endpoints.

 Important

1. Use either v1 or v2 endpoints. However, depending on which version you choose, the below step will be different. We recommend using v2 endpoints.


2. If you use v1 endpoints, add a body parameter named resource. For the value of this parameter, use Application ID of the back-end app.


3. If you use v2 endpoints, use the scope you created for the backend-app in the Default scope field. Also, make sure to set the value for the accessTokenAcceptedVersion property to 2 in your application manifest in Azure AD Client APP and Backend app.
   
   

    

   
   


4. Next, specify the client credentials. These are the credentials for the client-app.


5. For Client ID, use the Application ID of the client-app.
   
   

    

   
   


6. For Client secret, use the key you created for the client-app earlier.


7. Immediately following the client secret is the redirect_urls
   
   

    

   
   


8. Go back to your client-app registration in Azure Active Directory under Authentication.


9. .paste the redirect_url under Redirect URI, and  check the issuer tokens then click on Configure button to save.
   
   

    

   
   

Now that you have configured an OAuth 2.0 authorization server, the Developer Console can obtain access tokens from Azure AD.

The next step is to enable OAuth 2.0 user authorization for your API. This enables the Developer Console to know that it needs to obtain an access token on behalf of the user, before making calls to your API.

1. Go to APIs menu under the APIM


2. Select the API you want to protect and Go to Settings.


3. Under Security, choose OAuth 2.0, select the OAuth 2.0 server you configured earlier and select save.

Calling the API from the Developer Portal:

Now that the OAuth 2.0 user authorization is enabled on your API, the Developer Console will obtain an access token on behalf of the user, before calling the API.

1. Copy the developer portal url from the overview blade of apim
   
   

    

   
   


2. Browse to any operation under the API in the developer portal and select Try it. This brings you to the Developer Console.


3. Note a new item in the Authorization section, corresponding to the authorization server you just added.
   
   

    

   
   


4. Select Authorization code from the authorization drop-down list, and you are prompted to sign in to the Azure AD tenant. If you are already signed in with the account, you might not be prompted.
   
   

    

   
   


5. After successful sign-in, an Authorization header is added to the request, with an access token from Azure AD. The following is a sample token (Base64 encoded):
   
   

    

   
   


6. Select Send to call the API successfully with 200 ok response.
   
   

    

   
   

Validate-jwt policy to pre-authorize requests with AD token:

Why JWT VALIDATE TOKEN?

At this point we can call the APIs with the obtained bearer token.

However, what if someone calls your API without a token or with an invalid token? For example, try to call the API without the Authorization header, the call will still go through.

This is because the API Management does not validate the access token, It simply passes the Authorization header to the back-end API.

To pre-Authorize requests, we can use <validate-jwt> Policy by validating the access tokens of each incoming request. If a request does not have a valid token, API Management blocks it.

We will now configure the Validate JWT policy to pre-authorize requests in API Management, by validating the access tokens of each incoming request. If a request does not have a valid token, API Management blocks it.

1. Browser to the APIs from the left menu of APIM


2. Click on “ALL APIS” and open the inbound policy to add the validate-jwt policy(It checks the audience claim in an access token and returns an error message if the token is not valid.) and save it.
   
   

    

   
   


3. Go back to the developer portal and send the api with invalid token.


4. You would observe the 401 unauthorized.
   
   

    

   
   


5. Modify the token from authorization header to the valid token and send the api again to observe the 200-ok response.
   
   

Understanding <validate-jwt> Policy

In this section, we will be focusing on understanding how <validate-jwt> policy works (the image in the right side is the decoded JWT Token)

• The validate-jwt policy supports the validation of JWT tokens from the security viewpoint, It validates a JWT (JSON Web Token) passed via the HTTP Authorization header
  If the validation fails, a 401 code is returned.


•  The policy requires an openid-config endpoint to be specified via an openid-config element. API Management expects to browse this endpoint when evaluating the policy as it has information which is used internally to validate the token.
  Please Note : OpenID config URL differs for the v1 and v2 endpoints.


• The required-claims section contains a list of claims expected to be present on the token for it to be considered valid. The specified claim value in the policy must be present in the token for validation to succeed.

    The claim value should be the Application ID of the Registered Azure AD Backend-APP.

Reference Article : https://docs.microsoft.com/en-us/azure/api-management/api-management-access-restriction-policies#ValidateJWT

OAuth2 implicit grant flow:

The following diagram shows what the entire implicit sign-in flow looks like.

As mentioned, Implicit grant type is more suitable for the single page applications. In this grant type, The user is requested to signin by providing the user credentials

Once the credentials are validated the token is returned directly from the authorization endpoint instead of the token endpoint.

The token are short lived, and a fresh token will be obtained through a hidden request as user is already signed in.

NOTE : To successfully request an ID token and/or an access token, the app registration in the Azure portal – App registrations page must have the corresponding implicit grant flow enabled, by selecting ID tokens and access tokens in the Implicit grant and hybrid flows section. 

Implicit Flow – DEMO

The configuration for the implicit grant flow is similar to the authorization code, we would just need to change the Authorization Grant Type to “Implict Flow” in the OAuth2.0 tab in APIM as shown below.

After the OAuth 2.0 server configuration, The next step is to enable OAuth 2.0 user authorization for your API under APIs Blade :

Now that the OAuth 2.0 user authorization is enabled on your API, we can test the API operation in the Developer Portal for the Authorization type : “Implict”.

Once after choosing the Authorization type as Implicit, you should be prompted to sign into the Azure AD tenant. After successful sign-in, an Authorization header is added to the request, with an access token from Azure AD and APIs should successfully return the 200-ok response:

Client Credentials flow

The entire client credentials flow looks like the following diagram.

In the client credentials flow, permissions are granted directly to the application itself by an administrator.

Token endpoint is used to obtain a token using client ID and Client secret, the resource server receives the server and validates it before sending to the client.

Client Credentials – Demo

• In Client Credential flow, The OAuth2.0 configuration in APIM should have Authorization Grant Type as “Client Credentials”

• Specify the Authorization endpoint URL and Token endpoint URL with the tenant ID
  
  

   

  
  


• The value passed for the scope parameter in this request should be (application ID URI) of the backend app, affixed with the .default suffix : ”API://<Backend-APP ID>/.default”
  
  
  

   

  
  
  
  

Now that you have configured an OAuth 2.0 authorization server, The next step is to enable OAuth 2.0 user authorization for your API.

Now that the OAuth 2.0 user authorization is enabled on your API, we can test the API operation in the Developer Portal for the Authorization type : “Client Credentials”.

Once after choosing the Authorization type as Client Credentials in the Developer Portal,

1.  The sign in would happen internally with client secret and client ID without the user credentials.


2. After successful sign-in, an Authorization header is added to the request, with an access token from Azure AD.

1. Select Send to call the API successfully.

Detailing about Client Credential Flow:
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow

About .default scope : https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent#the-default-scope

Resource Owner Password Credentials

 The Resource Owner Password Credential (ROPC) flow allows an application to sign in users by directly handling their password.

The ROPC flow is a single request: it sends the client identification and user’s credentials to the Identity Provided, and then receives tokens in return.

The client must request the user’s email address and password before doing so. Immediately after a successful request, the client should securely release the user’s credentials from memory.

Resource Owner Password Credentials – Demo

Disclaimer: The new developer portal currently does not support the ROPC type and being worked upon by the Engineering team.

We will be covering the Demo in Legacy Developer Portal on ROPC as new portal does not support this type yet.

Please note that legacy portal is going to get expired in 2023.

The OAuth2.0 server configuration would be similar to the other grant types, we would need select the Authorization grant types as  Resource Owner Password :

You can also specify the Ad User Credentials in the Resource owner password credentials section:

Please note that it’s not a recommended flow as it requires a very high degree of trust in the application and carries risks which are not present in other grant types.

Now that you have configured an OAuth 2.0 authorization server, the next step is to enable OAuth 2.0 user authorization for your API.

Now that the OAuth 2.0 user authorization is enabled on your API, we will be browsing to the legacy developer portal and maneuver to the API operation

• Select Resource Owner Password from the authorization drop-down list


• You will get a popup to pass the credentials with the option to “use test user” if you check this option it will be allowing the portal to sign in the user by directly handling their password added during the Oauth2.0 configuration and generate the token after clicking on Authorize button :
  
  

   

  
  


• Another option is to uncheck the “test user” and Add the username and password to generate the token for different AD User and hit the authorize button
  
  

   

  
  


• The access token would be added using the credentials supplied:
  
  

   

  
  


• Select Send to call the API successfully.

Please note that the validate jwt policy should be configured for preauthorizing the request for Resource owner password credential flow also.

Things to remember

• The portal needs to be republished after API Management service configuration changes when updating the identity providers settings.


• Make sure to specify the correct Oauth Authorization & Token endpoint in OAuth2.0 configuration in APIM.
  For example :
  https://login.microsoftonline.com/<tenantID>/oauth2/v2.0/authorize
  
  https://login.microsoftonline.com/<tenantID>/oauth2/v2.0/token


• The Client App registration should have redirect url for the APIM developer portal
  
  https://<apim-name>.developer.azure-api.net/signin-oauth/implicit/callback
  
  =>https://<apim-name>.developer.azure-api.net/signin-oauth/code/callback/oauth-authorizationcodeflow

Common issues when OAuth2.0 is integrated with API Management: 

•  Problem faced while obtaining a token with Client Credential Grant Type:

       Error Snapshot:

Solution:

This error indicated that scope api://b29e6a33-9xxxxxxxxx/Files.Read is invalid.

As client_credentials flow requires application permission to work, but you may be passing the scope as Files.Read which is a delegated permission(user permission) and hence it rejected the scope.

To make it work, we would need to use default application scope as  “api://backendappID/.default”

   II.            Receiving “401 Unauthorized” response


 Solution:

You may be observing 401 unauthorized response returned by validate-jwt policy, its is recommended to look at the aud claims in the passed token and validate-jwt policy.

You can decode the token at  https://jwt.io/ and reverify it with the validate-jwt policy used in inbound section:
For example:

The Audience in the decoded token payload should match to the claim section of the validate-jwt policy:

<claim name=”aud”>

      <value>api://b293-9f6b-4165-xxxxxxxxxxx</value>

</claim>

•  Validate-JWT policy fails with IDX10511: Signature validation failed:
  
  

When we go to test the API and provide a JWT token in the Authorization header the policy may fail with the following error:  

  IDX10511: Signature validation failed. Keys tried: ‘Microsoft.IdentityModel.Tokens.X509SecurityKey , KeyId: CtTuhMJmD5M7DLdzD2v2x3QKSRY

Solution :
If you look at the metadata for the config url (https://login.microsoftonline.com/common/.well-known/openid-configuration)you will find a jwks_uri property inside the resulting json.

This uri will point to a set of certificates used to sign and validate the jwt’s.  You may find that the keyId (in this sample “CtTuhMJmD5M7DLdzD2v2x3QKSRY“) does exist there. 

Something like this:

{

“keys”: [{

“kty”: “RSA”,

“use”: “sig”,

“kid”: “CtTuhMJmD5M7DLdzD2v2x3QKSRY“,

“x5t”: “CtTuhMJmD5M7DLdzD2v2x3QKSRY”,

“n”: “18uZ3P3IgOySln……”,

“e”: “AQAB”,

“x5c”: [“MII…..”]

So it seems that it should be able to validate the signature.

If you look at the decoded jwt you may see something like this:

{

“typ”: “JWT”,

“alg”: “RS256”,

“x5t”: “CtTuhMJmD5M7DLdzD2v2x3QKSRY”,

“kid”: “CtTuhMJmD5M7DLdzD2v2x3QKSRY”

}

.{

“aud”: “00000003-0000-0000-c000-000000000000”,

“iss”: “https://sts.windows.net/<tenantID>/“,

“appid”: “1950a258-227b-4e31-a9cf-717495945fc2”,

“nonce”: “da3d8159-f9f6-4fa8-bbf8-9a2cd108a261”,

There’s a nonce in play here.  

This requires extra checking that validate-jwt does not do.  Getting a token for the Graph api and Sharepoint may emit a nonce property.  A token used to make calls to the Azure management api, however, will not have the nonce property.

The ‘nonce’ is a mechanism, that allows the receiver to determine if the token was forwarded. The signature is over the transformed nonce and requires special processing, so if you try and validate it directly, the signature validation will fail.

The validate jwt policy is not meant to validate tokens targeted for the Graph api or Sharepoint.  The best thing to do here is either remove the validate jwt policy and let the backend service validate it or use a token targeted for a different audience.

• Validate-JWT policy fails with IDX10205: Issuer validation failed

Here is an example configuration a user might have added to their policy:

<validate-jwt header-name=”Authorization” failed-validation-httpcode=”401″ failed-validation-error-message=”Unauthorized. Access token is missing or invalid.”>

    <openid-config url=”https://login.microsoftonline.com/72f988bf-86af-91ab-2d7cd011db47/.well-known/openid-configuration” />

    <required-claims>

       <claim name=”Aud” match=”any”>

           <value>api://72f988bf-86af-91ab-2d7cd011db47</value>

       </claim>

    </required-claims>

</validate-jwt>

When a we go to test that API and provide a JWT token in the Authorization header the policy may fail with the following error:

IDX10205: Issuer validation failed. Issuer: ‘https://login.microsoftonline.com/72f988bf-86af-91ab-2d7cd011db47/v2.0‘. Did not match: validationParameters.ValidIssuer: ” or validationParameters.ValidIssuers: ‘https://sts.windows.net/72f988bf-86af-91ab-2d7cd011db47/‘.

Solution:

This error message gets thrown when the Issuer (“iss”) claim in the JWT token does not match the trusted issuer in the policy configuration.

Azure Active Directory offers two versions of the token endpoint, to support two different implementations. AAD also exposes two different metadata documents to describe its endpoints. The OpenID Config files contains details about the AAD tenant endpoints and links to its signing key that APIM will use to verify the signature of the token. Here are the details of those two endpoints and documents (for the MSFT AAD tenant):

Azure AD Token Endpoint V1: https://login.microsoftonline.com/<tenantID>/oauth2/token

Azure AD OpenID Config V1: https://login.microsoftonline.com/<tenantID>/.well-known/openid-configuration

Azure AD Token Endpoint V2: https://login.microsoftonline.com/<tenantID>/oauth2/v2.0/token

Azure AD OpenID Config V2: https://login.microsoftonline.com/<tenantID>/v2.0/.well-known/openid-configuration

Error Details:

The error usually occurs because the user is using a mix between V1 and V2. So they request a token from V1 endpoint but configured <openid-config> setting pointing to V2 endpoint,  or vice versa.

The Azure AD V1 endpoint uses an issuer value of https://sts.windows.net/{tenant-id-guid}/

The Azure AD V2 endpoint uses an issuer value of https://login.microsoftonline.com/{tenant-id-guid}/v2.0

To resolve this issue you just need to make sure the <validate-jwt> policy is loading up the matching openid-config file to match the token. The easiest way is to just toggle the open-id config url within the policy and then it will move beyond this part of the validation logic.

<validate-jwt header-name=”Authorization” failed-validation-httpcode=”401″ failed-validation-error-message=”Unauthorized. Access token is missing or invalid.”>

    <openid-config url=”https://login.microsoftonline.com/{tenant-id-guid}/.well-known/openid-configuration” />

1. Find the <openid-config> setting in their policy


2. Just switch out the openid-config url between the two formats, replace {tenant-id-guid} with the Azure AD Tenant ID which you can collect from the Azure AD Overview tab within the Azure Portal

             https://login.microsoftonline.com/{tenant-id-guid}/.well-known/openid-configuration

             — or —-

             https://login.microsoftonline.com/{tenant-id-guid}/v2.0/.well-known/openid-configuration

How to Set Up Tailwind CSS in a SPFx Project

by Contributed | May 1, 2021 | Technology

This article is contributed. See the original author and article here.

In the State of CSS 2020 survey, the Tailwind CSS becomes the number 1 CSS Framework in terms of Satisfaction and Interest in the last 2 years. It also gets the awards for The Most Adopted Technology. It seems a lot of developers like this framework. Based on my experience, this framework can help us rapidly build UI by reducing complexity when styling the UI.

In this article, I will share my setup to use the Tailwind CSS in a SharePoint Framework (SPFx) project.

Prepare the SPFx Project

Prepare your SPFx project. I use a newly generated SPFx project (v1.11) but you can also use your existing SPFx project.

Install Modules

Install all modules needed by executing the command below:

npm install tailwindcss@1.9.6 postcss postcss-cli postcss-import @fullhuman/postcss-purgecss gulp-postcss autoprefixer@9.8.6 -D

Initialize Tailwind CSS and PostCSS

Initialize Tailwind CSS by executing the command below:

npx tailwind init -p –full

The command will create the tailwind.config.js in the project’s base directory. The file contains the configurations, such as colors, themes, media queries, and so on.

The command will also create the postcss.config.js file. We need PostCSS because we will use Tailwind CSS as a PostCSS plugin.

Inject Tailwind CSS Components and Utilities

We need to create a CSS file that will be used to import Tailwind CSS styles.

• Create an assets folder in the project’s base directory


• Create a tailwind.css file in the assets folder


• Add the following lines of code to the file:

@import “tailwindcss/components”;
@import “tailwindcss/utilities”;

Add Gulp Subtask for Processing Tailwind CSS

We need to add the Tailwind CSS build process to our SPFx build process.

• Open the gulpfile.js


• Add the following lines of code to the file (before the build.initialize(require(‘gulp’)); line):

const postcss = require(“gulp-postcss”);
const atimport = require(“postcss-import”);
const purgecss = require(“@fullhuman/postcss-purgecss”);
const tailwind = require(“tailwindcss”);

const tailwindcss = build.subTask(
   “tailwindcss”,
   function (gulp, buildOptions, done) {
      gulp
         .src(“assets/tailwind.css”)
         .pipe(
            postcss([
               atimport(),
               tailwind(“./tailwind.config.js”),
               …(buildOptions.args.ship
                  ? [
                       purgecss({
                          content: [“src/**/*.tsx”],
                          defaultExtractor: (content) =>
                             content.match(/[w-/:]+(?<!:)/g) || [],
                       }),
                    ]
                  : []),
            ])
         )
         .pipe(gulp.dest(“assets/dist”));
      done();
   }
);
build.rig.addPreBuildTask(tailwindcss);

The code will add the tailwindcss subtask to the SPFx Gulp Build task. It will also purge (remove unused styles) the Tailwind CSS for build with ship flag:

gulp build –ship

or

gulp bundle –ship

Add Reference to The Generated Tailwind CSS

We need to add reference the generated Tailwind CSS by adding the import code in your main .ts webpart file:

import ‘../../../assets/dist/tailwind.css’;

That’s it!

Now you can use Tailwind CSS utilities in your SPFx project.

Result

You might be familiar with the below result except it’s not using styles from the 74-lines scss/css file anymore.