This article is contributed. See the original author and article here.
Web activity is helpful when invoking an external endpoint from within an ADF pipeline. While Azure Data Factory/ Synapse pipelines offer various orchestration activities, web activity helps provide extensibility when building data pipelines that span across different services.
Web activities work exceptionally well with APIs that follow the asynchronous request-reply pattern, a suggested approach for building scalable web APIs/ services. In such scenarios, it automatically polls for the status endpoint (when it receives HTTP 202) till the API returns HTTP 200. It behaves synchronously and waits till the response is HTTP 200 before kicking off the down-stream activities.
Most Azure-based APIs are async request-reply pattern and gels with the existing web activity behavior.
But in case you have API endpoints that do not follow the above pattern and expect the caller to wait for a response till the processing/ actions are done. As default action, the web activity will timeout within 1 minute if it does not receive any response.
We are introducing a response timeout property ‘httpRequestTimeout’ in Web activity that will let you provide larger timeouts for such synchronous APIs.
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.