I've created a Pipeline that's executed by a trigger every time a blob is created, the problem is that are scenarios where the process needs to upload multiple files at the same time, when it happens, the pipeline executes as many times as the number of blobs and it causes that the data is wrong. I tried to cofigure a Copy Data Activity in the main Pipeline in order to copy every blob created, but since this pipeline is inside the first one, it executes many times as well.
The expected result would be that the pipeline just execute once (no matter how many blob are created) and the copy data activity, be able to copy all the blobs in the folder.
Here is my Pipeline and trigger configuration:
The Copy Data configuration down below:
I've been trying for months, and here is the other try I did:
Could you tell me what I'm doing wrong?
What you can do is filter the copy activity source based on the property Filter by last modified
, where you can specify a start time and end time in UTC.
you can try this Incrementally copy new and changed files based on LastModifiedDate by using the Copy Data tool
Here as per your scenario, just mention the Start time.
x
hours or to be safe in the last day based on how frequent you have the files created in Storage.triggerRunTimestamp
and append to a array variable.If this is feasible, I can spin an example pipeline.
Any reason why you are mapping your event trigger to the original path source where all files are being created and uploaded? Cant you create a dummy blob path at the end with a dummy file to have a final trigger once all files are uploaded to overcome this issue?
note: this is how we manage this:) but there is a redundant file generated unfortunately
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.