Pandas: use of json_normalize() with nested list of lists of dicts

Question

I tried figuring out a way of loading some data saved in a JSON format into a Pandas DataFrame using the function json_normalize(). The JSON file has the format:

data = [
            [{"v": [1, 2, 3]},
             {"x":
                    {"c": [1,1,1,1,1],
                     "w": [1,2,3,4]
                     },
              "f": 1,
              "b": [1,2,3,5]}
            ],
            [{"v": [4, 5, 6]},
             {"x":
                    {"c": [1,2,2,2,1],
                     "w": [1,2,3,4]
                     },
              "f": 0.07,
              "b": [7,2,5,7]}
            ]
        ]

Unfortunately I don't have the control over its format. I've tried everything my brain could come up using the meta and record_path .

I'd like to have a table with the columns ['v', 'f', 'b', 'c', 'w' ]. Clearly all columns except 'f' would be arrays.

Answer 1

You should just format your data like this:

for i in range(len(data)):
  _dict = {}
  for j in data[i]:
    for key,value in j.items():
      _dict[key] = value
  data[i] = _dict

Post that, simple json_normalize should work:

from pandas import json_normalize
result = json_normalize(data)
result.head()