Expanding List of Dicts dataframe

Question

Been trying for a while and unable to get the result I want.

Input:

"cursor":"YXJyYXljb25uZWN0aW9uOjA=",
"node":{
      "class":1,
      "end": [
          {"part": "a", "see" : "c"},
          {"part": "b", "see" : "c"}
       ]
}

Desired output in a DF.

cursor                     class       end.part    end.see            
YXJyYXljb25uZWN0aW9uOjA      1           a            c
YXJyYXljb25uZWN0aW9uOjA      1           b            c

I tried using json_normalise but couldn't get it working properly.

Any help would be greatly appreciated!

Thanks!

Answer 1

Here is another solution:

#!/usr/bin/python3
import pandas as pd

dic = {
    "cursor":"YXJyYXljb25uZWN0aW9uOjA=",
    "node":{
        "class":1,
        "end": [
            {"part": "a", "see" : "c"},
            {"part": "b", "see" : "c"}
        ]
    }
}

# 1. Use `meta` to specify what we want to show in the result
# 2. Use 'record_path` to flatten the nested list
df = pd.json_normalize(dic, meta=['cursor', ['node', 'class']], 
                       record_path=['node', 'end'] , record_prefix='end.')

# Rename the 'node.class' column to 'class'
df = df.rename(columns={'node.class': 'class'})

# Reindex by the order of columns list
df = df.reindex(columns=['cursor', 'class', 'end.part', 'end.see'])

print(df)

Reference of pd.json_normalise : https://towardsdatascience.com/all-pandas-json-normalize-you-should-know-for-flattening-json-13eae1dfb7dd

Answer 2

Try:

d = {
    "cursor": "YXJyYXljb25uZWN0aW9uOjA=",
    "node": {
        "class": 1,
        "end": [{"part": "a", "see": "c"}, {"part": "b", "see": "c"}],
    },
}

df = pd.json_normalize(d).explode("node.end")
x = df.pop("node.end").apply(pd.Series).add_prefix("end.")
df = pd.concat([df, x], axis=1)
print(df)

Prints:

                     cursor  node.class end.part end.see
0  YXJyYXljb25uZWN0aW9uOjA=           1        a       c
0  YXJyYXljb25uZWN0aW9uOjA=           1        b       c