This article is contributed. See the original author and article here.
Benefit of using session log in copy activity:
With session log enabled in copy activity, you can log your copied file names, which can help you to further ensure the data is not only successfully copied from source to destination store, but also consistent between source and destination store by verifying the copied file names with their metadata in the log files. When you enable fault tolerance setting in copy activity to skip faulty data, the skipped files and skipped rows can also be logged. For more information, you can get details from here.
How to get started:
You can create a pipeline with a copy activity, and then configure the copy activity to enable session log as below:
The sample log file is as below:
From the log file above, you can see sample1.csv has been skipped because it failed to be verified to be consistent between source and destination store. You can get more details about why sample1.csv becomes inconsistent is because it was being changed by other applications when ADF copy activity is copying at the same time. You can also see sample2.csv has been successfully copied from source to destination store.
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.