Azure Data Factory Pipeline 'On Failure'

Question

I am setting up an ADF pipeline to copy blob into an Azure SQL DB. I have a Iteration activity in my pipeline, where I have set up a counter to loop and copy only if the blob exists.

This works great except for some random PK violations, which I will have to check manually. So I edited my pipeline to log the error, and continue. So I set up the pipeline as such. If the copy activity fails due to Primary Key Violation, (for now) ignore, but log the details using a stored procedure and continue as usual ie update the loop counter to get the next folder.

Unfortunately, the success of Log Failure does not execute the "Set Variable" activity. So it goes back in an infinite loop keep coming back with the same exception, but the Stored Procedure activity itself is logging the error message correctly.

If I create a new "Set Variable" and do exactly what the SetLoopVariable does, it seems to be okay. but that means I have to copy every activity after that to have two separate paths. Which I feel is redundant.

BACKGROUND : My file structure is container/YYYY/MM/dd/HH/mm, there will be at least one file per hour, but not for every minute of the day so I to do a check if the folder exists before attempting to copy.

Answer 1

This is by design. SetVariable will only be called if Copy Data succeeds and fails, since Data Factory V2 Activity Dependencies are a Logical AND .

Answer 2

Thomas answer is correct. I had this exact issue recently. In case it helps someone else, I realised it means the arrows don't represent a flow but a dependency. The box only runs if all the preceding tasks are done, which is impossible in your case because it depends on the copy both succeeding and failing.

To solve your case just duplicate the 'set loop variable' in your error handling path.

However you might then have the same problem that I now have here Azure data factory: Handling inner failure in until/for activity