This article is contributed. See the original author and article here.
ADF has added a new option in the Azure Integration Runtime for data flow TTL: Quick re-use. This feature is currently available as a public preview. By selecting the re-use option with a TTL setting, you can direct ADF to maintain the Spark cluster for that period of time after your last data flow executes in a pipeline. This will provide much faster sequential executions using that same Azure IR in your data flow activities. You may find that parallel data flow executions or data flows that take have long execution times, will be better served as single job clusters without a TTL.
In the video below, I will walk you through how to use your TTL settings in the Azure IR so you can take your data flow sequential pipelines and move from 1-2 min warm pool startup times to activities that start in seconds:
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.