来自嵌套 JSON 的 Pandas 数据帧

Question

Given the following json dataset snapshot what is the best way to turn it into a pandas Data Frame?

Reading the file into a data frame ends up to which is not exactly usable.

I am currently using json_normalize to turn location and sensor to separate Data Frames

but trying the same approach with sensordatavalues gives me the following error Is this because sensordatavalues is an array object?

To make things worse, in some sensordatavalue records, the id key is missing

Just to make it a bit more challenging for you Pandas Gurus, Is there a way to do all the above in the same Data Frame?

        "location": {
            "indoor": 0,
            "exact_location": 0,
            "latitude": "37.36",
            "altitude": "17.0",
            "id": 13487,
            "country": "GL",
            "longitude": "26.962"
        },
        "sampling_rate": null,
        "id": 105462750,
        "sensordatavalues": [
            {
                "value_type": "temperature",
                "value": "18.70",
                "id": 226552256
            },
            {
                "value_type": "humidity",
                "value": "99.90",
                "id": 226552257
            }
        ],
        "sensor": {
            "id": 25666,
            "sensor_type": {
                "name": "DHT22",
                "id": 9,
                "manufacturer": "various"
            },
            "pin": "7"
        },
        "timestamp": "2020-01-19 19:10:38"
    },

Answer 1

Use pd.Series.explode to unpack the list into individual rows.

exploded = df['sensordatavalues'].explode()
exploded
# 0    {'value_type': 'temperature', 'value': '18.70'...
# 0    {'value_type': 'humidity', 'value': '99.90', '...
# Name: sensordatavalues, dtype: object

The index is maintained, with duplicates, so this can be easily rejoined to the original data.

joined = df.drop(columns='sensordatavalues').join(exploded)
joined
#                                             location sampling_rate         id                                             sensor            timestamp                                   sensordatavalues
# 0  {'indoor': 0, 'exact_location': 0, 'latitude':...          None  105462750  {'id': 25666, 'sensor_type': {'name': 'DHT22',...  2020-01-19 19:10:38  {'value_type': 'temperature', 'value': '18.70'...
# 0  {'indoor': 0, 'exact_location': 0, 'latitude':...          None  105462750  {'id': 25666, 'sensor_type': {'name': 'DHT22',...  2020-01-19 19:10:38  {'value_type': 'humidity', 'value': '99.90', '...

Then you can use json_normalize as before.

json_normalize(joined['sensordatavalues'])
#     value_type  value         id
# 0  temperature  18.70  226552256
# 1     humidity  99.90  226552257