How to log errors in dataflow adf of parallel sources

Question

I have to do some data engineering by reading manifest.cdm.json files from datalake. add pipeline run id column and push to sql database.

I have one json list file which have required parameter to read CDM json file in source of dataflow.

Previous Approach : I used Foreach and passed parameter to dataflow with single activity then error capturing. But use of Dataflow with for each costs too much..

Current Approch : I mannually created Dataflow with all cdm files. But here I'm not able to capture error. If any source got error all dataflow activity fails. and If I select skip error in dataflow activity I'm not getting any error.

So what should be the approch to get errors from current approch.

Answer 1

You can capture the error using set variable activity in Azure Data Factory.

Use below expression to capture the error message using Set Variable activity:

@activity('Data Flow1').Error.message

Later you can store the error message in blob storage for future reference using copy activity . In below example we are saving error message in .csv file using DelimitedText dataset.