Python stored dataframe from api(json)

Question

i need help with api(json) and store as dataframe. From json, i want only data in 'projects'. I have this function

  def get():
    url_address="https://xx.com/projects.json?limit=100&offset="+str('0')
    r = requests.request("GET",'https://xx.com/projects.json?limit=1',verify=False, auth=('xy','xy')).json()
    total_record=r['total_count']
    print("Total record: " +str(total_record))
   
    all_items = pd.DataFrame()


    for offset in range(0, total_record,100):
        url = "https://xx.com/projects.json?limit=100&offset="+str(offset)              
        response = requests.get(url=url,verify=False, auth=('xy','xy'))
        data=response.json()
        df = pd.DataFrame(data['projects'])
        all_items.append(df)
   
    all_items.to_csv("text.csv", encoding='utf-8', index=False)
        
        
 print(get())

But I have a problem with the append command because the resulting Dataframe is always empty. When I try it manually for one iteration without append, I get results. But this append probably doesn't do anything to me. Can anyone advise? Do I need a better way to do this? BTW: I have to do this like a cycle, because the API returns a maximum of 100 results.

Answer 1

Use a list instead of an empty dataframe then concatenate all items into a dataframe:

    all_items = []  # <- HERE

    for offset in range(0, total_record,100):
        url = "https://xx.com/projects.json?limit=100&offset="+str(offset)
        response = requests.get(url=url,verify=False, auth=('xy','xy'))
        data=response.json()
        df = pd.DataFrame(data['projects'])
        all_items.append(df)

    all_items = pd.concat(all_items)  # <- HERE
    all_items.to_csv("text.csv", encoding='utf-8', index=False)