Azure Data Factory -> Lookup , ForEach and copy activity

Question

I want to take a file from the azure synapse and load it in ADLs using ADf. I want to read the data of the Last 13 months and make a different file for each month. I made a CSV file where I wrote the start date and end date of each month and make a lookup activity over this file. Then using foreach activity, I load the file from the copy activity. Now I want to write a query for each month's data.

select * from edw.factbaldly where INSERT_DATE > @activity('Lookup1').output.value.startdate and INSERT_DATE < @activity('Lookup1').output.value.EndDate

select * from edw.factbaldly where INSERT_DATE > @item().startdate and INSERT_DATE < @item().EndDate'

I use these to queries but not able to read the data of lookup activity and fetch the data. Please help me with the query. Thanks in advance.

Answer 1

I assume your Lookup1 CSV column headings are startdate and enddate

In your ForEach > Settings > Items you will have @activity('Lookup1').output.value

Inside the ForEach block, your Copy activity Source query will look like:

select * from edw.factbaldly where INSERT_DATE > '@{item().startdate}' and INSERT_DATE < '@{item().enddate}'

ADF will substitute @{thing} with a string so you'll get the dates as quoted strings in the query

Maybe also you want one of the signs as >= or <=?

In fact you probably don't need to maintain the CSV because you can use a variable and ADF functions utcnow(), addToTime() and startOfMonth() to find the dates

Answer 2

In the lookup activity you will fetch the @item().startdate and @item().EndDate. Or I guess you have already set those in the lookup before ForEach. But to use this details when you produce new files, you must use the query from the question in source part of the Copy Activity.

If you can't use the query directly on the file, you can import the whole file to DB table and then use your query in the copy activity source.

Answer 3

You can use an expression like this

 @concat('select * from edw.factbaldly where INSERT_DATE> >',item().startdate,'and INSERT_DATE <',item().EndDate)

If I where you, i could have added a set variable activity and tested the above expression. The set variable should give us a syntactically correct TSQL statement. HTH