使用增量ID作为水印在Azure数据工厂管道中复制数据而不是日期时间

Question

I'm able to incrementally load data from an source Azure MSSQL DB to a sink Azure MSSQL DB using a timestamp. For some reason i wish to incrementally load the data using an incremental id in the source database instead of a timestamp. Is this possible?

I need to run the Copy Data activity only once a day. So i would also want to store the batch id for each of the copy data activity in a batch_details table.

I'm using ADF v.2

I'm a new to azure, How do i do it?

Answer 1

What you ask is basically what is explained in this tutorial from the official documentation: https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

You can use whatever you want as a watermark, the tutorial uses a datetime value, but an incremental id works too.

To run the pipeline once a day, use a trigger: https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#schedule-trigger

Hope this helped!