"如何使用 Python 请求对带有偏移参数的 api 调用进行分页"

Question

I need to make repeated api calls as there is a limit on each call of 1000 records. There are about 20,000 total records of which I test them , keep a sample, then need request the next 1000. The offset parameter is available.

p = getpass.getpass()
url = ("https://example.api.com/api/1.0/devices/all/?offset={}&include_cols=asset_no,name,service_level,building&type=physical" 
r = requests.get(url, auth=HTTPBasicAuth('admin', p))
data = json.loads(r.text)
payload = data["Devices"]

Answer 1

I have to keep this very general since you did not give further details and mentioned no API vendor.

Pagination can be done using a simple while loop.

The basic workflow is that while you are getting a pagination token in your response, keep making subsequent requests. In pseudo code that might look like this:

 Page = GetPageOfItems(); //process the data from the page, or add it to a larger array, etc. while( Page->cursor ) Page = GetPageOfItems(Page->cursor); //process the data again end

_{Ref: https://medium.com/square-corner-blog/tips-and-tricks-for-api-pagination-5cacc6f017da}

The implementation also depends on API details, eg does the data header contain the current offset and/or a hasMore key, eg

p = getpass.getpass()
offset=0

while True:
    url = ("https://example.api.com/api/1.0/devices/all/?offset=" + offset + "&include_cols=asset_no,name,service_level,building&type=physical" 
    r = requests.get(url, auth=HTTPBasicAuth('admin', p))
    data = json.loads(r.text)
    # Process the payload or add it to a list
    offset = data['offset'] # offset +1?
    hasMore = data['has-more']
    if not hasMore:
        break

Answer 2

In a general sense, pagination with limit and offset is a little formula :

number_of_offsets = total_items / limit_per_page

The offset parameter controls the starting point within the collection of resource results.

If you have a collection of 15 items to be retrieved from a resource and you specify limit=5 , you can retrieve the entire set of results in 3 successive requests by varying the offset value: offset=0 , offset=5 , and offset=10 .

The number offsets become the number of pages. So in our example above it would be :

offset = 0 : page 1
offset = 5 : page 2
offset = 10: page 3