[英]extract data from dictionary with nested dictionaries that contain lists that contain dictionaries
I have a response from an api that contains a data sump from a heating system, structured as dictionaries with nested dictionaries that contain lists that contain dictionaries.我有一个来自 api 的响应,其中包含来自加热系统的数据池,结构为带有嵌套字典的字典,其中包含包含字典的列表。
eg例如
sample = {"zoneType": "HEATING",
"interval": {"from": "2020-10-23T22:45:00.000Z", "to": "2020-10-24T23:15:00.000Z"},
"hoursInDay": 24,
"measuredData": {
"measuringDeviceConnected": {
"timeSeriesType": "dataIntervals",
"valueType": "boolean",
"dataIntervals": [{
"from": "2020-10-23T22:45:00.000Z", "to": "2020-10-24T23:15:00.000Z", "value": True}]
},
"insideTemperature": {
"timeSeriesType": "dataPoints",
"valueType": "temperature",
"min": {
"celsius": 19.34,
"fahrenheit": 66.81},
"max": {
"celsius": 20.6,
"fahrenheit": 69.08},
"dataPoints": [
{"timestamp": "2020-10-23T22:45:00.000Z", "value": {"celsius": 20.6, "fahrenheit": 69.08}},
{"timestamp": "2020-10-23T23:00:00.000Z", "value": {"celsius": 20.55, "fahrenheit": 68.99}},
{"timestamp": "2020-10-23T23:15:00.000Z", "value": {"celsius": 20.53, "fahrenheit": 68.95}},
{"timestamp": "2020-10-23T23:30:00.000Z", "value": {"celsius": 20.51, "fahrenheit": 68.92}},
{"timestamp": "2020-10-23T23:45:00.000Z", "value": {"celsius": 20.48, "fahrenheit": 68.86}},
{"timestamp": "2020-10-24T00:00:00.000Z", "value": {"celsius": 20.48, "fahrenheit": 68.86}},
{"timestamp": "2020-10-24T00:15:00.000Z", "value": {"celsius": 20.44, "fahrenheit": 68.79}}]
},
"humidity": {
"timeSeriesType": "dataPoints",
"valueType": "percentage",
"percentageUnit": "UNIT_INTERVAL",
"min": 0.615,
"max": 0.627,
"dataPoints": [
{"timestamp": "2020-10-23T22:45:00.000Z", "value": 0.615},
{"timestamp": "2020-10-23T23:00:00.000Z", "value": 0.615},
{"timestamp": "2020-10-23T23:15:00.000Z", "value": 0.619},
{"timestamp": "2020-10-23T23:30:00.000Z", "value": 0.620},
{"timestamp": "2020-10-23T23:45:00.000Z", "value": 0.621},
{"timestamp": "2020-10-24T00:00:00.000Z", "value": 0.623},
{"timestamp": "2020-10-24T00:15:00.000Z", "value": 0.627}]
}
}}
I want to extract the ['insideTemperature']['datapoints'] timestamp and celsius values from the above (actual data spans more periods) and place them as columns in a new pd.DataFrame along with other data from the 'humidity' key.我想从上面提取 ['insideTemperature']['datapoints'] 时间戳和 celsius 值(实际数据跨越更多时间段),并将它们作为列放在新的 pd.DataFrame 中以及来自“湿度”键的其他数据. In due course, I want to merge this with data from a separate API call that has a similar structure, though may not have consistent timestamp values.在适当的时候,我想将其与来自具有类似结构的单独 API 调用的数据合并,但可能没有一致的时间戳值。
A number of the top level dictionaries contain summary data (eg min and max values) so can be ignored.许多顶级词典包含汇总数据(例如最小值和最大值),因此可以忽略。 Equally, conversion from celsius to f etc, is easy to do if needed, so I don't want to pull this data.同样,如果需要,从 celsius 转换为 f 等很容易,所以我不想提取这些数据。
What is the best way to cleanly create a DataFile that lists the timestamp, temperature in Celsius and humidity from this query that I can then join with other query outputs?干净地创建一个 DataFile 的最佳方法是什么,该文件列出此查询中的时间戳、摄氏温度和湿度,然后我可以将其与其他查询输出连接?
So far, I have been using the following:到目前为止,我一直在使用以下内容:
import pandas as pd
df = pd.DataFrame(sample['measuredData']['insideTemperature']['dataPoints'])
## remove column that contains dictionary data, leaving time data
df.drop(labels='value', axis=1, inplace=True)
## get temp data into new column
input_data_point = sample['measuredData']['insideTemperature']['dataPoints']
temps = []
for i in input_data_point:
temps.append(i['value']['celsius'])
df['inside_temp_c'] = pd.DataFrame(temps)
## repeat for humidity
input_data_point = sample['measuredData']['humidity']['dataPoints']
temps = []
for i in input_data_point:
temps.append(i['value'])
df['humidity_pct'] = pd.DataFrame(temps)
Being new to coding in python, I am wondering if there is a far quicker way of extracting the data from the original downloaded data, straight into a clean Pandas DataFrame??作为 python 编码的新手,我想知道是否有一种更快的方法可以从原始下载的数据中提取数据,直接进入一个干净的 Pandas DataFrame? Grateful for any suggestions.感谢任何建议。
You can use json_normalize
to get the data:您可以使用json_normalize
来获取数据:
df1 = pd.json_normalize(sample,
record_path=['measuredData', 'insideTemperature', 'dataPoints'],
meta=['zoneType'])
print(df1)
df2 = pd.json_normalize(sample,
record_path=['measuredData', 'humidity', 'dataPoints'],
meta=['zoneType'])
print(df2)
df1: df1:
timestamp value.celsius value.fahrenheit zoneType
0 2020-10-23T22:45:00.000Z 20.60 69.08 HEATING
1 2020-10-23T23:00:00.000Z 20.55 68.99 HEATING
2 2020-10-23T23:15:00.000Z 20.53 68.95 HEATING
3 2020-10-23T23:30:00.000Z 20.51 68.92 HEATING
4 2020-10-23T23:45:00.000Z 20.48 68.86 HEATING
5 2020-10-24T00:00:00.000Z 20.48 68.86 HEATING
6 2020-10-24T00:15:00.000Z 20.44 68.79 HEATING
df2: df2:
timestamp value zoneType
0 2020-10-23T22:45:00.000Z 0.615 HEATING
1 2020-10-23T23:00:00.000Z 0.615 HEATING
2 2020-10-23T23:15:00.000Z 0.619 HEATING
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.