How to set and get variable value in Azure Synapse or Data Factory pipeline

Question

I have created a pipeline with Copy Activity , say, activity1 in Azure Synapse Analytics workspace that loads the following JSON to Azure Data Lake Storage Gen2 (ADLSGen2) using source as a REST Api and Sink (destination) as ADLSGen2 . Ref .

MyJsonFile.json (stored in ADLSGen2)

{"file_url":"https://files.testwebsite.com/Downloads/TimeStampFileName.zip"}

In the same pipeline, I need to add an activity2 that reads the URL from the above JSON , and activity3 that loads the zip file (mentioned in that URL) to the same Gen2 storage .

Question : How can we add an activity2 to the existing pipeline that will get the URL from the above JSON and then pass it to activity3 ? Or, are there any better suggestions/solutions to achieve this task.

Remarks : I have tried Set Variable Activity (shown below) by first declaring a variable in the pipeline and the using that variable, say, myURLVar in this activity, but I am not sure how to dynamically set the value of myURLVar to the value of the URL from the above JSON. Please NOTE the Json file name (MyJsonFile.json) is a constant, but zip file name in the URL is dynamic (based on timestamp), hence we cannot just hard code the above url.

Answer 1

As @Steve Zhao mentioned in the comments, use lookup activity to get the data from the JSON file and extract the required URL from the lookup output value using set variable activity.

• Connect the lookup activity to the sink dataset of previous copy data activity.

Output of lookup activity:

• I have used the substring function in set activity to extract the URL from the lookup output.

   @replace(substring(replace(replace(replace(string(activity('Lookup1').output.value),'"',''),'}',''),'{',''),indexof(replace(replace(replace(string(activity('Lookup1').output.value),'"',''),'}',''),'{',''),'http'),sub(length(string(replace(replace(replace(string(activity('Lookup1').output.value),'"',''),'}',''),'{',''))),indexof(replace(replace(replace(string(activity('Lookup1').output.value),'"',''),'}',''),'{',''),'http'))),']','')

Set variable output value :

Answer 2

There is a way to do this without needing complex string manipulation to parse the JSON. The caveat is that the JSON file needs to be formatted such that there are no line breaks (or that each line break represents a new record).

First setup a Lookup activity that loads the JSON file in the same way as @NiharikaMoola-MT's answer shows.

Then for the Set Variable activity's Value setting, use the following dynamic expression: @activity('<YourLookupActivityNameHere>').output.firstRow.file_url