Service Bus –Receive Message Asynchronously or synchronously?

by Contributed | Jan 18, 2021 | Technology

This article is contributed. See the original author and article here.

As we know there are two kinds of operations for program threads — Asynchronous and Synchronous. 

These are the definitions we can get from internet. 

Asynchronous operation means the process operates independently of other processes.  

Synchronous operation means the process runs only as a resource or some other process being completed or handed off.  

However, whether it’s good to Receive Message Asynchronously on the Azure Service Bus?  

Pre-requirement 

Before we start, please read these documents. Service Bus asynchronous messaging and Azure Service Bus messaging receive mode 

From the above pre–requisites, we learn the following: 

Azure Service Bus support both Asynchronous messaging patterns and Synchronous messaging patterns. Applications typically use asynchronous messaging patterns to enable several communication scenarios. 

This test is archived based on Service Bus PeekLock Receive mode. Here is more background information about the principle for PeekLock receive mode.  

The principle for PeekLock Receive mode is that: 

• Every time the Service Bus finds the next message to be consumed. 


• Locks it to prevent other consumers from receiving it. 


• Then, return the message to the application.  

There is a common exception for the Service Bus Lock expiration. This exception is because the message transaction time longer than Lock duration. It may be due to many reasons like Receive Application has high latency. This blog will also reproduce this Lock expired exception for Receive Messages Asynchronously and Synchronously. Let’s do a test now!  

Test Entities: 

I use a same Queue to do this test. The Max delivery count is 1.  If you are interested about the usage of “Max delivery count” please check from here Service Bus exceeding MaxDeliveryCount.  

Message lock duration time is 30s.  

My Program: 

Here I use different function in .Net for receive messages. All the functions have “Async” like ReceiveBatchAsync means the functions are working Asynchronously. 

To simulate the situation by sending a large number of messages, I use Batch function to receive 1000 messages at one time. 

• Here is the program that receives messages in Asynchronous patterns.  

using Microsoft.ServiceBus.Messaging; 

using System; 

using System.Collections.Generic; 

using System.Linq; 

using System.Text; 

using System.Threading.Tasks; 

namespace SendReceiveQueue 

{ 

    class Program 

    { 

        static string connectionString = “<your connection string>“; 

        static string queueName = “<queue name>“; 

        static void Main(string[] args) 

        { 

            MainAsync().GetAwaiter().GetResult(); 

        } 

        public static async Task MainAsync() 

        { 

            QueueClient receiveClient = QueueClient.CreateFromConnectionString(connectionString, queueName); 

            //create a sender on the queue 

            var Timestamp2 = new DateTimeOffset(DateTime.UtcNow).ToUnixTimeMilliseconds(); 

            Console.WriteLine(“Receiving message -, timestamp:{0}”, Timestamp2); 

             IEnumerable<BrokeredMessage> messageList = await receiveClient.ReceiveBatchAsync(1000); 

            foreach (BrokeredMessage message in messageList) 

            { 

                try 

                { 

                    var Timestamp0 = new DateTimeOffset(DateTime.UtcNow).ToUnixTimeMilliseconds(); 

                    Console.WriteLine(“Message”+message.GetBody<string>() +“time”+Timestamp0); 

                    await message.CompleteAsync(); 

                } 

                catch (Exception ex) 

                { 

                    var Timestamp3 = new DateTimeOffset(DateTime.UtcNow).ToUnixTimeMilliseconds(); 

                    Console.WriteLine(“abandon message – timestamp:{0},errorr message {1}”, Timestamp3,ex.Message); 

                    await message.AbandonAsync(); 

                } 

            } 

            await receiveClient.CloseAsync();  

        } 

    } 

} 

This is the result. The average time of receiving message is in 200ms to 300ms. 

• And this is the Code for receiving messages with Synchronous messaging patterns.  

using Microsoft.ServiceBus.Messaging; 

using System; 

using System.Collections.Generic; 

using System.Linq; 

using System.Text; 

namespace SendReceiveQueue 

{ 

    class Program 

    { 

        static string connectionString = “<your connection string>“; 

        static string queueName = “<queue name>“; 

        static void Main(string[] args) 

        { 

            MainTest(); 

        } 

        static void MainTest() 

        { 

            QueueClient receiveClient = QueueClient.CreateFromConnectionString(connectionString, queueName); 

            //create a sender on the queue 

            var Timestamp2 = new DateTimeOffset(DateTime.UtcNow).ToUnixTimeMilliseconds(); 

            Console.WriteLine(“Receiving message -, timestamp:{0}”, Timestamp2); 

            IEnumerable<BrokeredMessage> messageList = receiveClient.ReceiveBatch(1000); 

            foreach (BrokeredMessage message in messageList) 

            { 

                try 

                { 

                    var Timestamp0 = new DateTimeOffset(DateTime.UtcNow).ToUnixTimeMilliseconds(); 

                    Console.WriteLine(“Message” + message.GetBody<string>() + “time” + Timestamp0); 

                    message.Complete(); 

                } 

                catch (Exception ex) 

                { 

                    var Timestamp3 = new DateTimeOffset(DateTime.UtcNow).ToUnixTimeMilliseconds(); 

                    Console.WriteLine(“abandon message – timestamp:{0},errorr message {1}”, Timestamp3, ex.Message); 

                    message.Abandon(); 

                } 

            } 

            receiveClient.Close(); 

            Console.Read(); 

        } 

    } 

} 

This is the result. At first time the messages can also finish in 200ms to 300ms. But after a while It shows error for “lock expired”.  

Why didn’t we get any errors while using the Async pattern in this program? Why we got “Lock expired” exception while using Sync pattern?  

This exception is highly possible in receiving messages in batch function. Because all these 1000 messages were received in one operation. Using Peeklock receive mode, Service Bus locked all the 1000 messages at the same time. And then complete messages in Asynchronous patten, Messages can be completed without blocking. The await keyword provides a non-blocking way to start a task, then continue execution when that task is completed. It saved the Message complete time.  

But using Synchronous patten, all the Messages was completed one by one, the waiting time exceeds 30s. So, it shows “lock expired” error.  

You can get detailed information on how the asynchronous C# backend works from this document. Asynchronous programming in C# | Microsoft Docs

Test Result Summary 

• From the test result, it indicates that receiving messages with Asynchronous Messaging pattern would have higher ability than Synchronous Messaging pattern. We recommend using Asynchronous over than Synchronous. 

• However, if we receive a larger number of messages in one Batch operation like 10000 messages. This program with Asynchronous Messaging pattern also would get “lock expired” error. As mentioned before this “Lock expired” exception may be due to many reasons. That also the reason Service Bus have Dead Lettering Queue to prevent Service Bus message being lost. If you are interested in this topic, you are welcome to provide your comments.  

What’s new: Dedicated clusters for Azure Sentinel

by Contributed | Jan 18, 2021 | Technology

This article is contributed. See the original author and article here.

If you ingest over 1Tb a day into your Azure Sentinel workspace and/or have multiple Azure Sentinel workspaces in your Azure enrolment, you may want to consider migrating to a dedicated cluster, a recent addition to the deployment options for Azure Sentinel.

NOTE: Although this blog refers to a “dedicated cluster for Azure Sentinel”, the dedicated cluster being referred to is for Log Analytics, the underlying data store for Azure Sentinel. You may find official documents refer to Azure Monitor; Log Analytics is part of the wider Azure Monitor platform.

Overview

A dedicated cluster in Azure Sentinel does exactly what it says: you are given dedicated hardware in an Azure data center to run your Azure Sentinel instance. This enables several scenarios:

• Customer-managed Keys – Encrypt the cluster data using keys that are provided and controlled by the customer.


• Lockbox – Customers can control Microsoft support engineers access requests for data.


• Double encryption protects against a scenario where one of the encryption algorithms or keys may be compromised. In this case, the additional layer of encryption continues to protect your data.

Additionally, multiple Azure Sentinel workspaces can be added to a dedicated cluster. There are several advantages to using a dedicated cluster from a Sentinel perspective:

• Cross-workspace queries will run faster if all the workspaces involved in the query are added to the dedicated cluster. NB: It is still recommended to have as few workspaces as possible in your environment. A dedicated cluster still retains the limit of 100 workspaces that can be included in a single cross-workspace query. 


• All workspaces on the dedicated cluster share the Log Analytics capacity reservation set on the cluster(not the Sentinel capacity reservation), rather than having to have one Log Analytics capacity reservation per workspace which can allow for cost savings and efficiencies. NB: By enabling a dedicated cluster you commit to a capacity reservation in Log Analytics of 1Tb a day ingestion.

Considering migrating to a dedicated cluster?

There are some considerations and limitations for using dedicated clusters:

• 
  

  The max number of clusters per region and subscription is 2.

  
  


• All workspaces linked to a cluster must be in the same region.


• 
  

  The maximum of linked workspaces to cluster is 1000.

  
  


• 
  

  You can link a workspace to your cluster and then unlink it. The number of workspace link operations on particular workspace is limited to 2 in a period of 30 days.

  
  


• 
  

  Workspace link to cluster should be carried ONLY after you have verified that the Log Analytics cluster provisioning was completed. Data sent to your workspace prior to the completion will be dropped and won’t be recoverable.

  
  


• 
  

  Cluster move to another resource group or subscription isn’t supported at the time of writing this article.

  
  


• 
  

  Workspace link to cluster will fail if it is linked to another cluster.

  
  

The great news is that you can retrospectively migrate to a dedicated cluster, so if this feature looks like it would be useful to your organization, you can find more information and migration steps here. 

With thanks to @Ofer_Shezaf, @Javier Soriano and @Meir Mendelovich for their input into this blog post.

Accelerated Networking on HB, HC and HBv2

by Contributed | Jan 18, 2021 | Technology

This article is contributed. See the original author and article here.

Azure Accelerated Networking is now enabled on modern Azure HPC VM Sizes – HB, HC and HBv2. Accelerated Networking enables Single Root IO Virtualization (SR-IOV) for a VM’s SmartNIC resulting in throughput of up to 30 gigabits/sec, and lower and more consistent latencies.

Note that this enhanced Ethernet capability does not replace the dedicated InfiniBand network included with these modern HPC VM types. Instead, Accelerated Networking simply enhances the functions that the dedicated Ethernet interface has always performed such as loading OS images, connectivity to Azure Storage resources, or communicating with other resources in a customer’s VNET.

When enabling Accelerated Networking on supported Azure HPC VMs it is important to understand the changes you will see within the VM and what those changes may mean for your HPC workload:

A new network interface:

As the front-end ethernet NIC is now SR-IOV enabled, it will show up as a new network interface. For example, following is a screenshot from Azure HBv2 VM instance.

Figure 1: “mlx5_0” is the ethernet interface, and “mlx5_1” is the InfiniBand interface.

Figure 2: InfiniBand interface status (mlx5_1)

Changes to MPI launcher scripts:

Most MPIs do not need any changes to MPI launcher scripts. However, certain MPI libraries, especially those using UCX without a Libfabric layer, may need the InfiniBand device to be specified explicitly because of the additional network interface introduced with Accelerated Networking.

For example, OpenMPI using UCX (pml=ucx) fails by default as it tries to use the “mlx5_0” interface (i.e. the Accelerated Networking NIC) when what it requires is really the “mlx5_1” (i.e. the InfiniBand NIC).

This can be addressed by explicitly specifying the correct interface (UCX_NET_DEVICES=mlx5_1:1).

Below are the environment parameters for specifying the interface names for various MPI libraries:

MVAPICH2: MV2_IBA_HCA=mlx5_1 

OpenMPI & HPC-X: UCX_NET_DEVICES=mlx5_1:1

IntelMPI (mlx provider): UCX_NET_DEVICES=mlx5_1:1