Python中用于API的API分页

Question

I am using Python to extract data from a Solr API, like so:

import requests

user = 'my_username'
password= 'my password'
url = 'my_url'

print ("Accessing API..")
req = requests.get(url = url, auth=(user, password))
print ("Accessed!")
out = req.json()
#print(out)

However, it looks like in some of the API URLs: the output is fairly "large" (many of the columns are lists of dictionaries), and so it doesn't return all rows, which are necessary.

From looking around, it looks like I should use pagination to bring in results in specified increments. Something like this:

url = 'url?start=0&rows=1000'

Then,

 url = 'url?start=1000&rows=1000'

and so on, until there is no result is returned.

The way I am thinking about it is write a loop, and append result to output with every loop. However, I am not sure how to do that.

Would someone be able to help please?

Thank you in advance!

Answer 1

Did you look at the output? In my experience, solr response usually includes a 'numFound' in it's result. On a (old) solr I have locally, doing a random query. I get this result.

{
  "responseHeader": {
    "status": 0,
    "QTime": 1,
    "params": {
      "q": "*:*",
      "indent": "true",
      "start": "0",
      "rows": "10",
      "wt": "json",
      "_": "1509460751164"
    }
  },
  "response": {
    "numFound": 7023,
    "start": 0,
    "docs": [.. 10 docs]
    }
}

While working out this code example, I realized you don't need the numFound really. Solr will just return any empty list for docs if there are no further results. Making it easier to make the loop.

import requests

user = 'my_username'
password = 'my password'

# Starting values
start = 0
rows = 1000  # static, but easier to manipulate if it's a variable
base_url = 'my_url?rows={0}?start={1}'

url = base_url.format(rows, start)
req = requests.get(url=url, auth=(user, password))
out = req.json()

total_found = out.get('response', {}).get('numFound', 0)

# Up the start with 1000, so we fetch the next 1000
start += rows


results = out.get('response', {}).get('docs', [])
all_results = results

# Results will be an empty list if no more results are found
while results:
    # Rebuild url base on current start.
    url = base_url.format(rows, start)
    req = requests.get(url=url, auth=(user, password))
    out = req.json()
    results = out.get('response', {}).get('docs', [])
    all_results += results
    start += rows

# All results will now contains all the 'docs' of each request.
print(all_results)

Mind you.. those docs will be dict like, so more parsing will be needed.