Make multiple API calls - Python

Question

I often use an API call to pull some customer data. However, whenever I try to pull more than 20 customer ids, the API stops working. When this happens, I run multiple API calls, transform each JSON output into a df and append all the dataframes together.

That is fine when I need just a couple of API calls, but becomes inefficient when I have several customer ids to pull, as sometimes I have to run 5/10 separate API calls.

I thought a loop could help here. Given I have little experience with Python, I had a look at other questions on looping APIs, but I couldn't find a solution.

Below is the code I use. How can I make a single API call that loops through several customer ids (keeping in mind that there's a limit of circa 20 ids per call) and returns a single dataframe?

Thanks!

#list of customer ids
customer_id = [
"1004rca402itas8470der874",
"1004rca402itas8470der875,
"1004rca402itas8470der876",
"1004rca402itas8470der877",
"1004rca402itas8470der878",
"1004rca402itas8470der879"
]

#API call
payload = {'customer':",".join(customer_id), 'countries':'DE, 'granularity':'daily', 'start_date':'2021-01-01', 'end_date':'2022-03-31'}

response = requests.get('https://api.xxxxxxjxjx.com/t3/customers/xxxxxxxxxxxx?auth_token=xxxxxxxxxxxx', params=payload)

response.status_code

#convert to dataframe
api = response.json()
df = pd.DataFrame(api)
df['sales'] = df['domestic_sales'] + df['international_sales']
df = df[['customer_id','country','date','sales']]
df.head()

Answer 1

Here is the general idea:

# List of dataframes
dfs = []

# List of lists of 20 customer ids each
ids = [customer_id[i:i+20] for i in range(0, len(customer_id), 20)]

# Iterate on 'ids' to call api and store new df in list called 'dfs'
for chunk in ids:
    payload = {
        "customer": ",".join(chunk),
        "countries": "DE",
        "granularity": "daily",
        "start_date": "2021-01-01",
        "end_date": "2022-03-31",
    }
    response = requests.get(
        "https://api.xxxxxxjxjx.com/t3/customers/xxxxxxxxxxxx?auth_token=xxxxxxxxxxxx",
        params=payload,
    )
    dfs.append(pd.DataFrame(response.json()))

# Concat all dataframes
df = dfs[0]
for other_df in dfs[1:]:
    df = pd.concat([df, other_df])

# Additional work
df['sales'] = df['domestic_sales'] + df['international_sales']
df = df[['customer_id','country','date','sales']]