How to create DataFrame from json data - dicts, lists and arrays within an array

Question

I'm not able to get the data but only the headers from json data

Have tried to use json_normalize which creates a DataFrame from json data, but when I try to loop and append data the result is that I only get the headers.

import pandas as pd
import json
import requests
from pandas.io.json import json_normalize
import numpy as np

# importing json data

def get_json(file_path):
    r = requests.get('https://www.atg.se/services/racinginfo/v1/api/games/V75_2019-09-29_5_6')
    jsonResponse = r.json()
    with open(file_path, 'w', encoding='utf-8') as outfile:
        json.dump(jsonResponse, outfile, ensure_ascii=False, indent=None)

# Run the function and choose where to save the json file
get_json('../trav.json')

# Open the json file and print a list of the keys
with open('../trav.json', 'r') as json_data:
    d = json.load(json_data)

    print(list(d.keys()))

[Out]:
['@type', 'id', 'status', 'pools', 'races', 'currentVersion']

To get all data for the starts in one race I can use json_normalize function

race_1_starts = json_normalize(d['races'][0]['starts'])
race_1_starts_df = race_1_starts.drop('videos', axis=1)
print(race_1_starts_df)

[Out]:
    distance  driver.birth  ... result.prizeMoney  result.startNumber
0       1640          1984  ...             62500                   1
1       1640          1976  ...             11000                   2
2       1640          1968  ...               500                   3
3       1640          1953  ...            250000                   4
4       1640          1968  ...               500                   5
5       1640          1962  ...             18500                   6
6       1640          1961  ...              7000                   7
7       1640          1989  ...             31500                   8
8       1640          1960  ...               500                   9
9       1640          1954  ...               500                  10
10      1640          1977  ...            125000                  11
11      1640          1977  ...               500                  12

Above we get a DataFrame with data on all starts from one race. However, when I try to loop through all races in range in order to get data on all starts for all races, then I only get the headers from each race and not the data on starts for each race:

all_starts = []

for t in range(len(d['races'])):

    all_starts.append([t+1, json_normalize(d['races'][t]['starts'])])

all_starts_df = pd.DataFrame(all_starts, columns = ['race', 'starts'])
print(all_starts_df)

[Out]:
   race                                             starts
0     1      distance  ...                             ...
1     2      distance  ...                             ...
2     3      distance  ...                             ...
3     4      distance  ...                             ...
4     5      distance  ...                             ...
5     6      distance  ...                             ...
6     7      distance  ...                             ...

In output I want a DataFrame that is a merge of data on all starts from all races. Note that the number of columns can differ depending on which race, but that I expect in case one race has 21 columns and another has 20 columns - then the all_starts_df should contain all columns but in case a race do not have data for one column it should say 'NaN'.

Expected result:

[Out]:
race  distance  driver.birth  ... result.column_20     result.column_22
1       1640          1984  ...             12500                   1
1       1640          1976  ...             11000                   2
2       2140          1968  ...               NaN                   1
2       2140          1953  ...               NaN                   2
3       3360          1968  ...              1500                 NaN
3       3360          1953  ...            250000                 NaN

Answer 1

If you want all columns you can try this.. (I find a lot more than 20 columns so I might have something wrong.)

all_starts = []
headers = []
for idx, race in enumerate(d['races']):
    df = json_normalize(race['starts'])
    df['race'] = idx
    all_starts.append(df.drop('videos', axis=1))
    headers.append(set(df.columns))

# Create set of all columns for all races
columns = set.union(*headers)

# If columns are missing from one dataframe add it (as np.nan)
for df in all_starts:
    for c in columns - set(df.columns):
        df[c] = np.nan

# Concatenate all dataframes for each race to make one dataframe
df_all_starts = pd.concat(all_starts, axis=0, sort=True)

Alternatively, if you know the names of the columns you want to keep, try this

columns = ['race', 'distance', 'driver.birth', 'result.prizeMoney']
all_starts = []

for idx, race in enumerate(d['races']):
    df = json_normalize(race['starts'])
    df['race'] = idx
    all_starts.append(df[columns])

# Concatenate all dataframes for each race to make one dataframe
df_all_starts = pd.concat(all_starts, axis=0)