[英]Create a Pandas DataFrame from deeply nested JSON
I'm trying to create a single Pandas DataFrame object from a deeply nested JSON string.我正在尝试从深度嵌套的 JSON 字符串创建单个 Pandas DataFrame 对象。
The JSON schema is: JSON 架构是:
{"intervals": [
{
pivots: "Jane Smith",
"series": [
{
"interval_id": 0,
"p_value": 1
},
{
"interval_id": 1,
"p_value": 1.1162791357932633e-8
},
{
"interval_id": 2,
"p_value": 0.0000028675012051504467
}
],
},
{
"pivots": "Bob Smith",
"series": [
{
"interval_id": 0,
"p_value": 1
},
{
"interval_id": 1,
"p_value": 1.1162791357932633e-8
},
{
"interval_id": 2,
"p_value": 0.0000028675012051504467
}
]
}
]
}
Desired Outcome I need to flatten this to produce a table:期望的结果我需要将其展平以生成表格:
Actor Interval_id Interval_id Interval_id ...
Jane Smith 1 1.1162 0.00000 ...
Bob Smith 1 1.1162 0.00000 ...
The first column is the Pivots
values, and the remaining columns are the values of the keys interval_id
and p_value
stored in the list series
.第一列是
Pivots
值,其余列是存储在列表series
中的键interval_id
和p_value
的值。
So far i've got到目前为止我有
import requests as r
import pandas as pd
actor_data = r.get("url/to/data").json['data']['intervals']
df = pd.DataFrame(actor_data)
actor_data
is a list where the length is equal to the number of individuals ie pivots.values()
. actor_data
是一个长度等于个体数量的列表,即pivots.values()
。 The df object simply returns df 对象只是返回
<bound method DataFrame.describe of pivots Series
0 Jane Smith [{u'p_value': 1.0, u'interval_id': 0}, {u'p_va...
1 Bob Smith [{u'p_value': 1.0, u'interval_id': 0}, {u'p_va...
.
.
.
How can I iterate through that series
list to get to the dict values and create N distinct columns?如何遍历该
series
列表以获取 dict 值并创建 N 个不同的列? Should I try to create a DataFrame for the series
list, reshape it,and then do a column bind with the actor names?我应该尝试为
series
列表创建一个 DataFrame,对其进行整形,然后使用演员姓名进行列绑定吗?
UPDATE:更新:
pvalue_list = [i['p_value'] for i in json_data['series']]
this gives me a list of lists.这给了我一个列表列表。 Now I need to figure out how to add each list as a row in a DataFrame.
现在我需要弄清楚如何将每个列表添加为 DataFrame 中的一行。
value_list = []
for i in pvalue_list:
pvs = [j['p_value'] for j in i]
value_list = value_list.append(pvs)
return value_list
This returns a NoneType这将返回一个 NoneType
Solution解决方案
def get_hypthesis_data():
raw_data = r.get("/url/to/data").json()['data']
actor_dict = {}
for actor_series in raw_data['intervals']:
actor = actor_series['pivots']
p_values = []
for interval in actor_series['series']:
p_values.append(interval['p_value'])
actor_dict[actor] = p_values
return pd.DataFrame(actor_dict).T
This returns the correct DataFrame.这将返回正确的 DataFrame。 I transposed it so the individuals were rows and not columns.
我调换了它,所以个人是行而不是列。
I think organizing your data in way that yields repeating column names is only going to create headaches for you later on down the road.我认为以产生重复列名的方式组织你的数据只会让你以后头疼。 A better approach IMHO is to create a column for each of
pivots
, interval_id
, and p_value
.一种更好的方法是IMHO创建的每一个的柱
pivots
, interval_id
,和p_value
。 This will make extremely easy to query your data after loading it into pandas.在将数据加载到 Pandas 后,这将使查询数据变得非常容易。
Also, your JSON has some errors in it.此外,您的 JSON 中有一些错误。 I ran it through this to find the errors.
我通过它来查找错误。
import sh
jq = sh.jq.bake('-M') # disable colorizing
json_data = "from above"
rule = """[{pivots: .intervals[].pivots,
interval_id: .intervals[].series[].interval_id,
p_value: .intervals[].series[].p_value}]"""
out = jq(rule, _in=json_data).stdout
res = pd.DataFrame(json.loads(out))
This will yield output similar to这将产生类似于
interval_id p_value pivots
32 2 2.867501e-06 Jane Smith
33 2 1.000000e+00 Jane Smith
34 2 1.116279e-08 Jane Smith
35 2 2.867501e-06 Jane Smith
36 0 1.000000e+00 Bob Smith
37 0 1.116279e-08 Bob Smith
38 0 2.867501e-06 Bob Smith
39 0 1.000000e+00 Bob Smith
40 0 1.116279e-08 Bob Smith
41 0 2.867501e-06 Bob Smith
42 1 1.000000e+00 Bob Smith
43 1 1.116279e-08 Bob Smith
Adapted from this comment改编自此评论
Of course, you can always call res.drop_duplicates()
to remove the duplicate rows.当然,您始终可以调用
res.drop_duplicates()
来删除重复的行。 This gives这给
In [175]: res.drop_duplicates()
Out[175]:
interval_id p_value pivots
0 0 1.000000e+00 Jane Smith
1 0 1.116279e-08 Jane Smith
2 0 2.867501e-06 Jane Smith
6 1 1.000000e+00 Jane Smith
7 1 1.116279e-08 Jane Smith
8 1 2.867501e-06 Jane Smith
12 2 1.000000e+00 Jane Smith
13 2 1.116279e-08 Jane Smith
14 2 2.867501e-06 Jane Smith
36 0 1.000000e+00 Bob Smith
37 0 1.116279e-08 Bob Smith
38 0 2.867501e-06 Bob Smith
42 1 1.000000e+00 Bob Smith
43 1 1.116279e-08 Bob Smith
44 1 2.867501e-06 Bob Smith
48 2 1.000000e+00 Bob Smith
49 2 1.116279e-08 Bob Smith
50 2 2.867501e-06 Bob Smith
[18 rows x 3 columns]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.