Merging dataset results in an Azure Data Factory pipeline

Question

I am reading a JSON-formatted blob from Azure Storage. I am then using one of the values in that JSON to query a database to get more information. What I need to do is take the JSON from the blob, add the fields from the database to it, then write that combined JSON to another Azure Storage. I cannot, however, figure out how to combine the two pieces of information.

I have tried custom mapping in the copy activity for the pipeline. I have tried parameterized datasets, etc. Nothing seems to provide the results I'm looking for.

Is there a way to accomplish this using native activities and parameters (ie not by writing a simple utility and executing it as a custom activity)?

Answer 1

For this I would recommend create a custom U-SQL job to do what you want. So first lookup for both the data you want. Do the job in the U-SQL job and copy the results to the Azure Storage. See this example for your pipeline:

If you are not familiar to U-SQL this can help you:

• https://saveenr.gitbooks.io/usql-tutorial/content/

Also this will help you working with Json in your job:

• https://www.taygan.co/blog/2018/01/09/azure-data-lake-series-working-with-json-part-2

• https://www.taygan.co/blog/2018/03/02/azure-data-lake-series-working-with-json-part-3