Azure 数据工厂 - REST API 调用分页

Question

I'm making a call for data in Data Factory and struggling to call the url in the "next_page" item.

This is an example of what the first API call returns:

{
"items": [
    {
        "title_one": "TTL-55924",
        "id": "CPT-TTL-64577_TTL-55924",
        "title_id": "TTL-64577"
    },
    {
        "title_one": "TTL-69015",
        "id": "CPT-TTL-79755_TTL-69015",
        "title_id": "TTL-79755"
    }
],
"next_page": "http://api.com/api/info?offset=5000&key=XXXxxxXXXxxx"

}

I'm not sure which options to use in the Pagination Rules of my Copy activity.

Currently I'm trying the option "AbsoluteURL" with the value "$['next_page']" but this just returns an error.

Answer 1

If your API response contains the next page URL property, then the “ AbsoluteUrl “ pagination rule is the correct option to load the next page in the Azure data factory.

The supported values for pagination rules are mentioned in this MS document .

As mentioned in the example from the above document, Facebook Graph API returns the response as,

{
"data": [
    …
    …
],
"paging": {
    …
    …
    },
    "previous": "https://graph.facebook.com/me/albums?limit=25&before=NDMyNzQyODI3OTQw",
    "next": "https://graph.facebook.com/me/albums?limit=25&after=MTAxNTExOTQ1MjAwNzI5NDE="
}
}

Note : Pagination value of a JSON path expression starts with “ $ ”.

Your pagination in REST copy activity looks like this:

In your API, the pagination should look like

Answer 2

I have a similar issue. I am trying to copy shopify data from the rest API and was able to get the 1st page of data, however i cannot figure out how to set the pagination. THe pagination is coming in the header response like this:

I have it set to this but this returns an error:

ADF ERROR: Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 404 indicates failure. Request URL: https://joyfolie.myshopify.com/admin/api/2021-07/%3Chttps://joyfolie.myshopify.com/admin/api/2021-07/products.json?limit=50&page_info=eyJsYXN0X2lkIjo0NjI0ODcyMjEwNDkwLCJsYXN0X3ZhbHVlIjoiKk5FVyogQmFlIFNraXJ0IGluIE9jaHJlIiwiZGlyZWN0aW9uIjoibmV4dCJ9%3E;%20rel=%22next%22 Response payload:{"errors":"Not Found"},Source=Microsoft.DataTransfer.ClientLibrary,' Source Pipeline Copy Shopify Products

THis is how the link looks in the response header

https://joyfolie.myshopify.com/admin/api/2022-01/products.json?limit=50&page_info=eyJsYXN0X2lkIjo0NjI0ODcyMjEwNDkwLCJsYXN0X3ZhbHVlIjoiKk5FVyogQmFlIFNraXJ0IGluIE9jaHJlIiwiZGlyZWN0aW9uIjoibmV4dCJ9 ; rel="next"