How to process only the failed files in for each activity when the master pipeline is retriggered in ADF

Question

There are two pipelines Master and child. In the child pipeline there is a foreach activity which takes files as input and process them in parallel. For instance, there are 4 files, in which 2 files are successfully processed and loaded the data into a table. Then, 3rd file processing is failed and 4th file processing is successful. Now, when I retrigger the Master pipeline I only want the 3rd file to be processed, not all the 4 files. How can we achieve this.

I have tried below.

To move/delete the file once the processing is completed

But as per the requirement, I should not move/delete the file. Could someone please assist.

Answer 1

I create a test and succefuly achieve that. My overall idea is: use the Lookup activity to extract the copied file names array from the sql table, and then do a Filter operation with the source file names array. If the file name already exists in the sql table, the file copy activity will not be performed. It needs us to add file name to the sql table in Copy activity via Aditional columns .

In my sql table, it looks like as follows:

1. I declared 3 variables. arr1 Array type variable stores source file names. filterArray Array type variable stores copied file names array from the sql table.

2. At lookup activity, we can use this query select distinct FileName from [dbo].[emp] to get copied file names array from the sql table.

3. Assign the value to the variable filterArray .

4. I set the default value ["emp.csv","emp2.csv","emp3.csv","emp4.csv"] as source file names to the variable arr1 .

5. At Foreah acivity, we can foreach the variable arr1 .

6. Inside Foreach activity, assign the value @item() to the variable arrItem .

7. Then do Filter operation. Items: @variables('filterArr') , Condition: @contains(item().FileName,variables('arrItem')) , This item() here represents each element in the filterArray array.

8. At If condition activity, use @empty(activity('Filter1').output.Value) to determine whether this file has been copied.

9. In Ture activity, key in dynamic content @item() , this represents the name of the file to be copied.

That's all.