Getting dates for historical data copy in Azure datafactory

Question

I have to copy historical data from a rest api for one year eg., from March 1,2019 to March1,2020.The Rest API takes a start and end date as params.

To prevent load, however I have to call the API in pieces like copy with the start and end date as March1,2019 to March 30,2019....Once thats done then April 1,2019 to april 30,2019 and so on till March 1,2020 automatically and without manual intervention.

I was able to use utc now and add days for copying data for previous day to current startofday but am unable to figure out the copy of historical data.Any idea if this is possible?

Answer 1

You can try something like this:

1.create two variable named start_date and end_date.

2.create a Until activity, and type this expression: @greaterOrEquals(variables('end_date'),'2020-03-01T00:00:00Z')

3.create a Copy activity

Source setting:

Source dataset:

Sink dataset:

4.create tow Set variable activity to change start_date and end_date

Result:

By the way, you can change your date format according you need. Reference https://docs.microsoft.com/en-us/dotnet/standard/base-types/standard-date-and-time-format-strings .