Unpack dictionary from Pandas Column

Question

I have a dataframe that has one of the columns as a dictionary. I want to unpack it into multiple columns (ie code, amount are separate columns in the below Raw column format). The following code used to work with pandas v0.22, now (0.23) giving an index error:

pd.DataFrame.from_records(df.col_name.fillna(pd.Series([{'code':'not applicable'}], index=df.index)).values.tolist())

ValueError: Length of passed values is 1, index implies x

I searched google/stack overflow for hours and none of the other solutions previously presented work anymore.

Raw column format:

     dict_codes
0   {'code': 'xx', 'amount': '10.00',...
1   {'code': 'yy', 'amount': '20.00'...
2   {'code': 'bb', 'amount': '30.00'...
3   {'code': 'aa', 'amount': '40.00'...
10  {'code': 'zz', 'amount': '50.00'...
11                            NaN
12                            NaN
13                            NaN

Does anyone have any suggestions?

Thanks

Answer 1

Setup

df = pd.DataFrame(dict(
    codes=[
        {'amount': 12, 'code': 'a'},
        {'amount': 19, 'code': 'x'},
        {'amount': 37, 'code': 'm'},
        np.nan,
        np.nan,
        np.nan,
    ]
))

df

                         codes
0  {'amount': 12, 'code': 'a'}
1  {'amount': 19, 'code': 'x'}
2  {'amount': 37, 'code': 'm'}
3                          NaN
4                          NaN
5                          NaN

---

apply with pd.Series

Make sure to dropna first

df.codes.dropna().apply(pd.Series)

   amount code
0      12    a
1      19    x
2      37    m

---

df.drop('codes', 1).assign(**df.codes.dropna().apply(pd.Series))

   amount code
0    12.0    a
1    19.0    x
2    37.0    m
3     NaN  NaN
4     NaN  NaN
5     NaN  NaN

---

tolist and from_records

Same idea but skip the apply

pd.DataFrame.from_records(df.codes.dropna().tolist())

   amount code
0      12    a
1      19    x
2      37    m

---

df.drop('codes', 1).assign(**pd.DataFrame.from_records(df.codes.dropna().tolist()))

   amount code
0    12.0    a
1    19.0    x
2    37.0    m
3     NaN  NaN
4     NaN  NaN
5     NaN  NaN

Answer 2

Setup

                        codes
0  {'amount': 12, 'code': 10}
1    {'amount': 3, 'code': 3}

apply with pd.Series

df.codes.apply(pd.Series)

   amount  code
0      12    10
1       3     3