Azure 数据工厂中没有上次修改日期和主键字段的增量加载

Question

I am trying to do incremental load in azure data factory. Most of the tables in the database doesn't have last modified date column. I don't have rights to add watermark columns in the tables. Is there any way to do incremental loading without last modified date and primary key column?

I don't know which approach I can use. so kindly help me. Thanks in advance.

Answer 1

If you source database support native Change Data Capture feature, then you can use ADF Mapping data flow with no timestamp or ID columns are required to identify the changes since it uses the native change data capture technology in the databases.

For complete demonstration, please refer to this public documentation: Change data capture in Azure Data Factory and Azure Synapse Analytics

Another possible approach is if you can access both old data (Previously loaded data to your sink) and latest data (source) with changes then you can use mapping data flow in ADF and implement hashing to compare the both the datasets and pick the changed data as per your requirement.

You can refer to this demonstration for the same implementation: Data Flows: How to capture changed data