如何从 ADF 复制数据活动中获取给定列的最大值

Question

I have a copy data activity for on-premise SQL Server as source and ADLS Gen2 as sink. There is a control table to pickup tableName, watermarkDateColumn and the watermarkDatetime to pull incremental data from the source database.

After data is pulled/loaded in sink, I want to get the max of the watermarkDateColumn in my dataset. Can it be obtained from @activity('copyActivity1').output ?

I'm not allowed to use one extra lookup activity to query the source table for getting the max(watermarkDateColumn) in pipeline.

Answer 1

Copy activity only could be used for data transmission,not for any other aggregation feature. So @activity('copyActivity1').output won't help. Since you said you can't use lookup activity, i'm afraid your requirement is not available so far.

If you prefer not using additional activities, I suggest you using Data Flow Activity instead which is more flexible.There is built-in aggregation feature in the Data Flow Activity.