This article is contributed. See the original author and article here.
When you move data from source to destination store, Azure Data Factory copy activity provides an option for you to do additional data consistency verification to ensure the data is not only successfully copied from source to destination store, but also verified to be consistent between source and destination store. Once inconsistent files have been found during the data movement, you can either abort the copy activity or continue to copy the rest by enabling fault tolerance setting to skip inconsistent files. You can get the copied and skipped file names by enabling session log setting in copy activity.
How data consistency verification works:
With data consistency verification enabled, when copying binary files, ADF copy activity will verify file size, lastModifiedDate, and MD5 checksum for each binary file copied from source to destination store to ensure the data consistency between source and destination store. When copying tabular data, copy activity will verify the row count after job completes. Be aware the copy performance will be affected by enabling this option.
In ADF copy activity, you can choose to enable “Data consistency verification” setting, and then decide if you want to abort the copy activity job or continue to copy the rest once inconsistent files found as below:
Get the verification result after copy job completes:
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.