Convert this nested JSON to pandas dataframe

Question

Here is my json:

my_json = {
    "machine_name": {
        "0100": [
            {
                "date": "21/03/2019",
                "chainage": "27760.156",
                "unix_time": "1553110535",
                "time": "03:35:35",
                "f0001": "0.0",
                "f0002": "0.0",
                "f0006": "0.0",
                "f0007": "0.0",
                "f0008": "0.0",
                "f0009": "0.0"
            },
            {
                "date": "22/03/2019",
                "chainage": "27760.156",
                "unix_time": "1553110535",
                "time": "03:35:35",
                "f0001": "0.0",
                "f0002": "0.0",
                "f0006": "0.0",
                "f0007": "0.0",
                "f0008": "0.0",
                "f0009": "0.0"
            }
        ],
        "0101": [
            {
                "date": "21/03/2019",
                "chainage": "27761.498",
                "unix_time": "1553131029",
                "time": "09:17:09",
                "f0001": "0.347",
                "f0002": "0.007",
                "f0006": "2.524",
                "f0007": "0.0",
                "f0008": "121.036",
                "f0009": "0.0"
            },
            {
                "date": "22/03/2019",
                "chainage": "27761.498",
                "unix_time": "1553131029",
                "time": "09:17:09",
                "f0001": "0.347",
                "f0002": "0.007",
                "f0006": "2.524",
                "f0007": "0.0",
                "f0008": "121.036",
                "f0009": "0.0"
            }
        ]
    }
}

I want to create a pandas dataframe with the headers "date","chainage","unix_time", etc... merging the 'array of objects' for "0100" and "0101" into a single dataframe.

I have looked at read_json and json_normalize, but the outputs are not what expected. Any ideas how to achieve the desired result?

Answer 1

>>> rows = [v[0] for k, v in my_json['machine_name'].items()]
>>> rows # I fixed up the line-wrapping here for readability.
[{'date': '21/03/2019', 'chainage': '27760.156', 'unix_time': '1553110535', 
'time': '03:35:35', 'f0001': '0.0', 'f0002': '0.0', 'f0006': '0.0', 
'f0007': '0.0', 'f0008': '0.0', 'f0009': '0.0'}, {'date': '21/03/2019',
'chainage': '27761.498', 'unix_time': '1553131029', 'time': '09:17:09',
'f0001': '0.347', 'f0002': '0.007', 'f0006': '2.524', 'f0007': '0.0',
'f0008': '121.036', 'f0009': '0.0'}]

This gave us a list of the actual dicts wrapped inside single-element lists that are the dict values under machine_name , which we can then make a table from normally:

>>> df = pd.DataFrame(rows)

and add the index:

# we need to convert to Index explicitly from the dict_keys.
>>> index = pd.Index(my_json['machine_name'].keys())
>>> df.set_index(index, inplace=True)

The result looks right to me:

>>> df
       chainage        date  f0001  f0002  ...    f0008 f0009      time   unix_time
0100  27760.156  21/03/2019    0.0    0.0  ...      0.0   0.0  03:35:35  1553110535
0101  27761.498  21/03/2019  0.347  0.007  ...  121.036   0.0  09:17:09  1553131029

[2 rows x 10 columns]

Answer 2

the following seems to work.

One note, given the way you are declaring your my_json variable, Python will read it as a dictionary, not as a json string.

import pandas as pd


my_json = { ... } # you data --it will be read as a dictionary by default.

# for convenience, create this variable
d = my_json['machine_name']


# create a list to store each row (i.e.: 0100, 0101)
dict_ls = []

# loop through d and store each internal dictionary (i.e: d[0100], d[0101]...etc) in the list
for row in d.keys():
  dict_ls.append(d[row][0])

# convert the list of dictionaries into a dataframe
df = pd.DataFrame(dict_ls)