Incremental load in Azure Data Factory

Question

I am replicating my data from Azure SQl DB TO Azure SQL DB. I have some tables with date columns and some tables with just the ID columns which are assigning primary key. While performing incremental load in ADF, I can select date as watermark column for the tables which have date column and id as watermark column for the tables which has id column, But the issue is my id has guid values, So can I i take that as my watermark column? and if yes while copy activity process it gives me following error in ADF

Please see the image for above reference

How can I overcome this issue. Help is appreciated

Thank you Gp

I have tried dynamic mapping https://martinschoombee.com/2022/03/22/dynamic-column-mapping-in-azure-data-factory/ from here but it does not work it still gives me same error.

Answer 1

Regarding your question about watermak: A watermark is a column that has the last updated time stamp or an incrementing key So GUID column would not be a good fit. Try to find a date column, or an integer identity which is ever incrementing, to use as watermark. Since your source is SQL server, you can also use change data capture. Links: Incremental loading in ADF Change data capture

Regards, Chen

Answer 2

The watermark logic takes advantange of the fact that all the new records which are inserted after the last watermark saved should only be considered for copying from source A to B, basically we are using ">=" operator to our advantage here.

In case of guid you cannot use that logic as guid cann surely be unique but not ">=" or "=<" will not work.