Converting A Dictionary Of Lists That Contain Dictionaries To A Dataframe

Question

I have researched and tested ways to do this for a full day now and while I have found some topics that were of some help, nothing so far as worked 100%. I have found information in the Python documentation and on Stack Overflow that talk about how to convert a dictionary of dictionaries to a dataframe, but the data structure I am working with has significant differences with the data structures in those examples. Let me begin with describing where my data is coming from and what it looks like.

I am receiving the data from the public API of an online chess site. The data is in a JSON file and contains information about members of the site and is broken down by how active the member has been (Weekly, Monthly, and All_Time), and includes the members user name and the date he joined. Below is an example of how that data is structured:

 {
  "weekly": [
    {
        "username": "string", //username
        "joined": "integer",  //timestamp
    }
  ],
  "monthly": [
    {
        "username": "string", //username
        "joined": "integer",  //timestamp
    }
  ],
  "all_time": [
    {
        "username": "string", //username
        "joined": "integer",  //timestamp
    }
  ]
}

My goal is to get this data into a Pandas df in Jupyter Notebook so I can create graphs showing how active the members are.

Clearly, using pd.DataFrame.from_dict(data) will raise an error. I need to parse the data and write it to a file in a format that Pandas can handle. So far, my code is writing the username and joined data to a file called members. There are two things I need to work out. 1. I need to include the high level key data (Weeks, Months, and all_time) and include it in my members file. Each Key should show up only once in the file and be followed by all the user data for that catagory (ie. who has been active in that period). 2. I need to work out how to format the data in the file so Pandas can put it in a df. At the moment, the username and join data are written to the file with spaces as the deliminator. Here is the code. It runs without error so if you want, you can run it.

import requests 
import json



def getPlayerNames():


    headers = {
        'User-Agent': ' @Knightburgler/1.0 (Python 3.x)',
    'Accept-encoding': 'gzip'
    }


    url = 'https://api.chess.com/pub/club/team-iowa/members'
    response = requests.get(url, headers)

    response.raise_for_status()
    data = response.json()

    fp = open('members.txt', 'w')
    for period in data["weekly"], data["monthly"], data["all_time"]:
        for k in period:
            fp.write(k['username'] + " " + str(k['joined']) + " ")
    fp.close()

def main():
    getPlayerNames()


if __name__ == "__main__":
    main()

# eof

Once the data is written to a CSV file, the format should look like this:

username, joined date, period

Answer 1

You can create a DataFrame using a list of dictionaries. As follows

rows = []
for period in data:
  for k in data[period]:
    rows.append({'username': k['username'], 'joined':k['joined'], 'period': period})

df = pd.DataFrame(rows)

Once you have a DataFrame. Saving it is as easy as:

df.to_csv(filename)