Trouble with GET request to Copy Data from REST API to Data Lake

Question

I will provide some context: my pipeline makes a GET Request to a REST API (Auth type: OAuth2 Client Credential) in order to import data to the Data Lake (ADLSGen2) in parquet file format. Later, a Stored Procedure creates a View which includes every file in a predefined directory.

I am looking forward to requesting data to the API on an hourly basis (or maybe every 30 minutes) in order to get information of the previous hour. The thing is: almost 36 million records are brought per hour as a response.

In the body of the response there is no reference to the number or the total of pages. There is only data (keys and values).

On the other hand, the Headers include "first-page" and "next-page" (this one appears only if there are further pages in the response, but also makes no reference to the total of pages).

I was wondering if there are any useful suggestions to make my Copy Data activity work differently. Right now, and because of what I mentioned above, the pagination rule is set to RFC5988. I would like my requested data to be partitioned in some way.

Also, I was wondering if there is another way to approach this issue (like using another activity, for example).

Thanks!

Mateo

Answer 1

You need to replace the Header placeholder with your header_name(Link).

Or you can directly use like this dynamic content.