Unpacking list of lists of dicts column in Pandas dataframe

Question

I have df_in where one of the columns is a list of lists of dicts :

df_in = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [
        [{'B1': 1, 'B2': 2, 'B3': 3}, {'B1': 4, 'B2': 5, 'B3': 6}, {'B1': 7, 'B2': 8, 'B3': 9}],
        [{'B1': 10, 'B2': 11, 'B3': 12}],
        [{'B1': 13, 'B2': 14, 'B3': 15}, {'B1': 16, 'B2': 17, 'B3': 18}]
    ],
    'C': ['a', 'b', 'c']
})

df_in
    A   B                                                   C
0   1   [{'B1': 1, 'B2': 2, 'B3': 3}, {'B1': 4, 'B2': ...   a
1   2   [{'B1': 10, 'B2': 11, 'B3': 12}]                    b
2   3   [{'B1': 13, 'B2': 14, 'B3': 15}, {'B1': 16, 'B...   c

What I want to achieve is a general approach to unpacking B so that (1) every unique key ( B1 , B2 and B3 in this case) gets put into a column. And (2) stack mutiple lists in each row as new observations. I think an example output explains this best:

df_out = pd.DataFrame({
    'A': [1, 1, 1, 2, 3, 3],
    'B1': [1, 4, 7, 10, 13, 16],
    'B2': [2, 5, 8, 11, 14, 17],
    'B3': [3, 6, 9, 12, 15, 18],
    'C': ['a', 'a', 'a', 'b', 'c', 'c']
})

df_out
    A   B1  B2  B3  C
0   1   1   2   3   a
1   1   4   5   6   a
2   1   7   8   9   a
3   2   10  11  12  b
4   3   13  14  15  c
5   3   16  17  18  c

Any ideas?

Answer 1

Use dictionary comprehension with concat and DataFrame.pop for extract column:

df1 = pd.concat({k: pd.DataFrame(x) for k, x in df_in.pop('B').items()})
print (df1)
     B1  B2  B3
0 0   1   2   3
  1   4   5   6
  2   7   8   9
1 0  10  11  12
2 0  13  14  15
  1  16  17  18

Add original data by DataFrame.join and for correct order extract and append C column:

df = df_in.join(df1.reset_index(level=1, drop=True)).reset_index(drop=True)
df['C'] = df.pop('C')
print (df)
   A  B1  B2  B3  C
0  1   1   2   3  a
1  1   4   5   6  a
2  1   7   8   9  a
3  2  10  11  12  b
4  3  13  14  15  c
5  3  16  17  18  c

Alternative solution with DataFrame.assign , for correct order is used DataFrame.insert :

df1 = pd.concat([pd.DataFrame(v['B']).assign(A=v['A'], C=v['C']) 
                   for k, v in df_in.to_dict('index').items()], ignore_index=True)
df1.insert(0, 'A', df1.pop('A'))
print (df1)
   A  B1  B2  B3  C
0  1   1   2   3  a
1  1   4   5   6  a
2  1   7   8   9  a
3  2  10  11  12  b
4  3  13  14  15  c
5  3  16  17  18  c

Answer 2

You can try df.explode to unstack the list of dicts, then read the dicts as a dataframe and add to df_in :

df_in = df_in.explode('B')

df_out = pd.concat([df_in.reset_index(drop=True), 
                    pd.DataFrame(df_in['B'].tolist())], 
                    axis=1).drop('B', axis = 1).sort_index(axis=1)

df_out

   A  B1  B2  B3  C
0  1   1   2   3  a
1  1   4   5   6  a
2  1   7   8   9  a
3  2  10  11  12  b
4  3  13  14  15  c
5  3  16  17  18  c