Azure 数据工厂仅从 Blob 存储复制数据新添加的文件

Question

I would like to create a pipeline with Copy Data (mapping JSON source to MySQL columns). At this moment it is working, but the tool is copying all the files from my source container after the 'created' trigger . Is it possible to retrieve only 'new added' files in my container, so I do not have double data in MySQL?

Example current situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 1 + file 2

Example expected situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 2

Answer 1

This is actually a very used scenario , you should use LastModifiedDate for copying most recently added files and to prevent duplication.

Please check this link : https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview