How to correctly fetch data from API endpoint with multiple parameter in python?

Question

I tried to fetch data from API endpoints which has multiple parameters and build big pandas dataframe from it. In my attempt, I passed the root URL and iterate the thought API endpoint's multiple parameters with a customized values list. My goal is to fetch the data from specific API endpoints but passing different values to its multiple values and finally generate one pandas dataframe. Currently, my code returned me an empty dataframe. I am afraid I might do something wrong in my attempt. Can anybody point me out what went wrong in my code? Does anyone suggest a possible way of doing this in pandas? Any thoughts?

my current attempt

import pandas as pd
import json, requests

commodityCodes = [101, 102, 103, 104, 105]
countyCodes = [1220, 2010, 2050, 2110, 3330, 3370, 5210, 5460, 5490, 5520]
marketyear= list(range(2010,2020))

api_key = 'ec95a478-e46e-47f9-b57d-3d19012d527d'
rooturl = 'https://apps.fas.usda.gov/OpenData/api/esr/exports/'
headers = {'API_KEY': '{key}'.format(key=api_key)}

finaldf=pd.DataFrame()

for cc1 in commodityCodes:
    for cc2 in countyCodes:
        for year in marketyear:
            jsonData = requests.get(rooturl+'commodityCode/{}'.format(cc1)+ '/countryCode/{}'.format(cc2)+'/marketyear/{}'.format(year), headers=headers).json()
            df= pd.read_json(json.dumps(jsonData))
    finaldf.append(df)

but my above code returned me an empty dataframe. Maybe I used a nested loop to do this and that might cause the problem. Also running the above code is takes some minutes.

What would be an efficient way of doing this? Where was I wrong in my attempt? How can I correctly fetch data from API endpoints? Can anyone suggest a possible way of doing this? Any ideas?

Answer 1

Two issues:

1. Pandas' append returns a new object, it doesn't modify the original dataframe
2. you append in the outer loop, not in the inner loop

---

finaldf = pd.DataFrame()

for cc1 in commodityCodes:
    for cc2 in countyCodes:
        for year in marketyear:
            jsonData = requests.get(rooturl+'commodityCode/{}'.format(cc1)+ '/countryCode/{}'.format(cc2)+'/marketyear/{}'.format(year), headers=headers).json()
            df = pd.read_json(json.dumps(jsonData))
            finaldf = finaldf.append(df)

Suggestion

Build your url in one step, it's more readable

rooturl = 'https://apps.fas.usda.gov/OpenData/api/esr/exports/commodityCode/{}/countryCode/{}/marketyear/{}'
url = rooturl.format(cc1, cc2, year)