如何使用Python从Web API收集所有结果？

Question

I am collecting data from a web API by using a Python script. The web API provides maximum 50 results ( "size":50 ). However, I need to collect all the results. Please let me know how can I do it. My initial code is available below. Thank you in advance.

def getData():
    headers = {
    'Content-type': 'application/json',
    }

    data = '{"size":50,"sites.recruitment_status":"ACTIVE", "sites.org_state_or_province":"VA"}'
    response = requests.post('https://clinicaltrialsapi.cancer.gov/v1/clinical-trials', headers=headers, data=data)

    print(response.json())

Answer 1

Everything is in the doc :

https://clinicaltrialsapi.cancer.gov/#!/Clinical45trials/searchTrialsByGet

GET clinical-trials

Filters all clinical trials based upon supplied filter params. Filter params may be any of the fields in the schema as well as any of the following params...

size: limit the amount of results a supplied amount (default is 10, max is 50)

from: start the results from a supplied starting point (default is 0)

...

So you just have to specify a "from" value, and increment it 50 by 50.

Answer 2

To add to the answer already given you can get then total results from the initial json. You can then use a loop to increment for batches

import requests
import json

url = "https://clinicaltrialsapi.cancer.gov/v1/clinical-trials"
r = requests.get(url).json()
num_results = int(r['total'])
results_per_request = 50
total = 0
while total < num_results:
    total+=results_per_request
    print(total)