如何使用 Azure 数据工厂对日期或键列执行增量加载

Question

I wanted to achieve an incremental load from oracle to Azure SQL data warehouse using azure data factory. The Issue that I am facing is I don't have any date column or any key column to perform Incremental load Is there any other way to achieve this.

Answer 1

You will either have to:

A. Identify a field in each table you want to use to determine if the row has changed B. Implement some kind of change capture feature on the source data

Those are really the only the only two ways to limit the amount of data you pull from the source.

It wouldn't be very efficient, but if you are just trying not to update rows that haven't changed in your destination, you can hash your source values and hash the values in the destination, and only insert/update rows where the hashes don't match. Here's an example of how this works in T-SQL.

There is a section of the Data Factory documentation dedicated to incrementally loading data. Please check it out if you haven't.