Azure 数据工厂管道 - 如果内部活动失败

Question

I have developed an Azure Data Factory pipeline that checks the schema of files in the data lake. I have 15 files in the "templates format" and I check them against files received from customers. I have both xlsx and CSV files.

First I take all file names from the template folder and iterate them. I need to check if the current file is excel, as when I need to define whether a file is an xlsx or CSV.

I want to handle both types of issues. First when a schema of the received file is different from the schema of the template. And second when I have not received this file at all. So in the Set Variable activity, I use this code. So I can use it in switch activity.

@concat('Excel',
if(equals(activity('Get Metadata of excel bronze').Status,'Succeeded'),' True' , ' False'),
if(equals(activity('Get Metadata of excel bronze').Status,activity('Get Metadata of excel 
template').Status),' True' , ' False')
)

However, when I have not received a file and I have a template "Get a metadata of excel" the template fails and the whole "Is Excel" activity fails. Is there a possibility to force if the activity to not fail on error? An error is expected in this pipeline.

Answer 1

You can add a new activity upon failure so that even Get Metadata fails, pipeline keeps running the next steps.

Note : When you are connecting 2 different activity outputs to a new activity (set variable activity) as input, the Set Variable activity only runs when both inputs are successful and fails to run if any one of the input activities is failed.

The workaround is to connect the Get Metadata outputs to 2 different activities.

For each Get Metadata activity, add set variable activity (or any other required activity) on success and failure individually to run the pipeline without stopping when error occur.

Pipeline runs even when Get Metadata2 fails when connected to other activity upon failure.