Understanding Pipeline Failures and Error Handling

by Scott Muniz | Sep 3, 2020 | Uncategorized

This article is contributed. See the original author and article here.

Azure Data Factory orchestration allows conditional logic and enables user to take different based upon outcomes of a previous activity. In total we allows four conditional paths: Upon Success (default pass), Upon Failure, Upon Completion, and Upon Skip. Using different paths allow users to build robust pipelines and incorporates error handling in their ETL/ELT logic.

Here are two common error handling pattern we see customers use:

1. TRY-CATCH block. Define the business logic, and only defines Upon Failure path to catch any error from previous activities 
2. DO-IF-ELSE block. Define the business logic, and depends on the outcome of the activity, enacts either Upon Success path or Upon Failure path

Both are valid ways to incorporate error handling into the pipeline. However, upon pipeline execution, they may show different outcomes. Approach #1, TRY-CATCH, shows pipeline succeeds if Upon Failure path clears, where as approach #2, DO-IF-ELSE show pipeline failed if Upon Failure path is enacted.

Technical reasons for the difference is that, Azure Data Factory defines pipeline success and failures as follows:

• Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead
• Pipeline result is success if and only if all leaves succeed

Applying the logic to previous examples. 

1. In approach #1 TRY-CATCH block:
   • when previous activity succeeds: the node activity, Upon Failure, is skipped and its parent node succeeds, so overall pipeline succeeds
   • when previous activity fails: the node activity, Upon Failure, enacted and overall pipeline succeeds if Upon Failure path succeeds
2. In approach #2 DO-IF-ELSE block:
   • when previous activity succeeds: one node activity, Upon Success, succeeded, and the other node activity, Upon Failure, is skipped and its parent node succeeds; so overall pipeline succeeds
   • when previous activity fails: one node activity, Upon Success, is skipped and its parent node failed; so overall pipeline failed

Here is a table summarizing the difference

Service Bus – duplicate detection for partitioned messaging entity

by Scott Muniz | Sep 2, 2020 | Uncategorized

This article is contributed. See the original author and article here.

Pre-requirements:

Before we start, please read these document about Duplicate Detection, Partitioned queues and topics and Message Sessions 

From the above Pre-requirements, we learn the followings

• Enabling Duplicate detection helps to ensure that a duplicate message, which has the same message Id cannot be sent into a messaging entity during a specified time.

• Service Bus Partitions enable queues and topics or message entities to be partitioned across multiple message brokers and messaging stores. Enable partitioning the overall throughput will separate to different partition. Partition key can be used in some scenarios, such as sessions or transactions, require messages to be stored in a specific partition.

• Microsoft Service Bus Session enable joint and ordered handling of unbounded sequences of messages.  There are two patterns of it , first out and request-response pattern. Any sender can create a session when submitting messages into a topic or queue by setting the SessionId property to some application-defined identifier that is unique to the session.

So, from these above knowledges, we know that Azure Service Bus Queue and Topic/Subscription can enable for Duplicate Detection, Partitions and Sessions. But what’s the relationship between them?

From the meaning of Duplication detection, we know that there is a MessageId of each message. Definition of MessageId is shown below

“If the queue or topic has the RequiresDuplicateDetection property set to true and the SessionId or PartitionKey properties are not set, then the MessageId property value serves as the partition key”

This means the duplicate detection only can work when SessionID or PartitionKey are not be set. Is that a correct statement? Let’s do a test!

Test Entities:

Queues:

There are three queue I used, testqueue, testqueue2, testqueue3. At first time all of them have 0 messages.

• testqueue:  Enable Batched Operation, Requires Duplicate Detection. And set duplicate detection for 10 mins.

• testqueue2: Enable Batched Operation, Requires Duplicate Detection, Requires Session

• testqueue3: Enable Batched Operation, Requires Duplicate Detection, Requires Session, Enable Partition

My Program:

Here is the code. The different in this program used in each time, just the BrokeredMessage SessionID and PartitionKey that in red color.

The Session ID here set with a random value. From the above document, when enable both Partition key and Session ID, the value of them need to be same. Otherwise Service Bus returns an invalid operation exception.

    class Program

    {

        static string connectionString = [ConnectionString];

        static string queueName = “testqueue3”;// testqueue,testqueue2

        static void Main(string[] args)

        {

            MainAsync();

        }

        static void MainAsync()

        {

            QueueClient sendClient = QueueClient.CreateFromConnectionString(connectionString, queueName);

            //create a sender on the queue

            var session = Guid.NewGuid().ToString();

            var partitionKey = session;

            sendClient.SendBatch(Enumerable.Range(0, 10).Select(n =>

            {

                Console.WriteLine(“test my message n{0}”, session);

                BrokeredMessage test = new BrokeredMessage(“Hello World!”) { SessionId = session, , PartitionKey = session, MessageId = “deadbeef-dead-beef-dead-beef” + n };

                return test;

            }

            ));

        }

}

Test Round 1:

To find the message, I sent 10 message each time first.

• testqueue (Enable Batched Operation, Requires Duplicate Detection):  send 10 messages first, then immediately  send another same 10 messages, it still displays 10 messages.

• testqueue2(Enable Batched Operation, Requires Duplicate Detection, Requires Session): send 10 messages first, and send another same 10 messages, here also display 10 messages.

• testqueue3(Enable Batched Operation, Requires Duplicate Detection, Requires Session, Enable Partition): Add new message twice the message count turn to 20.

You can see in the list there are message with duplicate MessageId.

Then I checked for same MessageId “deadbeef-dead-beef-dead-beef0” , and I found they have  different Partition Keys.

Here are these three queues result in this test.

The above test about messages sent in duplicate detection duration. How about we wait for 10 mins to check after the duplicate detection time?

Test Round 2

After 10 mins, I sent 1000 messages again twice. The result shows, testqueue and testqueue2 have 1000 included the 10 messages sent before. For testqueue3, there are 2020 messages.

Test Result Summary

• From the first round test result, it indicates that messages with same MessageId can be detected as duplication in a non-partitioned entity within specified duplicate detection duration (10 mins).

• However with partitioned entity, duplicate messages which has same MessageId may not be detected as duplication due to they can end up in a different partition. In conclusion, messages can only be detected as duplication within the specified duplicate detection duration when they are landed in the same partition, i.e. same partition key

• For the second test with 1000 message were sent twice, it indicates that the duplicate duration detect only works for the specified duration. After the duplicate detection period, the Messages with same MessageId can be sent to the same Service Bus messaging entity.