Pandas：如何将 dicts 列表中的 dicts 列表展平到数据框中，如果嵌套列表中的任何 dict 缺少任何指定的键，则会抛出错误？

Question

My ultimate goal is to flatten a key within a list of dicts into a dataframe. The key's value is also a list of dicts, and that list could be empty for any given record in the top-level list. I'm trying to do this quickly, so trying to use vectorized operations in pandas, including json_normalize (which I assume is better than a loop).

In the resulting dataframe, I want to keep some top-level columns while flattening all of the keys in the list of nested dicts. I also want the operation to fail if there is any dict in any nested list that does not have all of the keys I specify. But, I do not want the operation to fail if the key exists but is None . (This is why I can't just check NaN after the normalization -- None is converted to NaN for float types in json_normalize , since it does not provide a dtype arg, so I wouldn't know whether it was NaN because the key didn't exist, or because it did exist but was None ).

For example, I tried doing something like this:

data = [
  {
    'id': 1, 
    'topfield1': "1-1",
    'topfield2': "1-2",
    'topfield3': "1-3",
    'topfield4': "1-4",
    'payments': [
      {'id': 1, 'amt': 2.0, 'not_required': 'something'},
      {'id': 2, 'amt': 4.0}
    ]
  },
  {
    'id': 2, 
    'topfield1': "2-1",
    'topfield2': "2-2",
    'topfield3': "2-3",
    'topfield4': "2-4",
    'payments': [
      {'id': 1}
    ]
  },
  {
    'id': 3, 
    'topfield1': "3-1",
    'topfield2': "3-2",
    'topfield3': "3-3",
    'topfield4': "3-4",
    'payments': []
  }
]

# now flatten into one row for each item in each record's 'payments' key, keeping top-level 'id', 'topfield1', 'topfield4' and raising error if payments.id or payments.amt does not exist in a payment

#ideally, it would work like this:

# i want this to raise an error since data[1]['payments'][0] does not have key 'amt'. apparently that's not how json_normalize works -- it just throws a TypeError because apparently meta columns can't be from the record_path
pandas.io.json.json_normalize(data, record_path='payments', meta=['id', 'topfield4', 'topfield1', ['payments', 'id'], ['payments', 'amt']]) 
'''expected output
KeyError: 'payments.amt' or something like that
'''
'''actual output
TypeError: list indices must be integers or slices, not str
'''

data[1]['payments'][0]['amt'] = None
# now every 'payment' has 'id' and 'amt' keys so should succeed. but this still throws TypeError.
pandas.io.json.json_normalize(data, record_path='payments', meta=['id', 'topfield4', 'topfield1', ['payments', 'id'], ['payments', 'amt']]) 
'''expected output
id  topfield4  topfield1  payments.id  payments.amt  payments.not_required
1      1-4     1-1             1            2.0            something
1      1-4     1-1             2            4.0              NaN
2      2-4     2-1             1            NaN              NaN
'''
'''actual output
TypeError: list indices must be integers or slices, not str
'''

But this doesn't work. Whenever I use fields from the payments objects -- ie keys of the objects in the record_path list -- as fields in the meta arg, I get TypeError: list indices must be integers or slices, not str .

It also seems like json_normalize isn't smart enough to actually follow the paths you give it in the meta arg, since: (though I guess I could just rename the relevant top-level columns to avoid)

# data as from above
pandas.io.json.json_normalize(data, record_path='payments', meta=['id'])
# fails with ValueError: Conflicting metadata name id, need distinguishing prefix`
# (shouldn't it know `id` is top-level since it's not a nested path?)

Is there a vectorized/fast way to accomplish what I want to do?

Thanks for any help.

EDIT 1: There could be many top level fields and I want a subset of them in the final dataframe. EDIT 2: Added more fields to make a minimum reproduceable example.

Answer 1

IIUC, we can flatten your dict with a simple comprehension and read the outer id as the keys to use as the index. we then concat whilst reading the values as a dataframe.

df = pd.concat(
    {d["id"]: pd.DataFrame.from_dict(d["payments"]) for d in data}, sort=False, axis=0
).reset_index(0).rename(columns={"id": "payments.id", "level_0": "id"})

---

print(df)

   id  payments.id  amt not_required
0   1          1.0  2.0    something
1   1          2.0  4.0          NaN
0   2          1.0  NaN          NaN