Convert pandas series with list of dictonaries to dataframe with columns from dictonary

Question

I've a pandas series with list of dictonaries:

series = pd.Series(
    [[{'id': '1', 'val': 'ab'}, {'id': '2', 'val': 'abc'}], [{'id': '1', 'val': 'aa'}, {'id': '2', 'val': 'ewe'}],
     [{'id': '3', 'val': 'aea'}, {'id': '4', 'val': 'te'}]],
    index=['2014-01-01 22:59:00+00:00', '2014-01-02 22:59:00+00:00', '2014-01-03 21:59:00+00:00'])

2014-01-01 22:59:00+00:00    [{'id': '1', 'val': 'ab'}, {'id': '2', 'val': 'abc'}]
2014-01-02 22:59:00+00:00    [{'id': '1', 'val': 'aa'}, {'id': '2', 'val': 'ewe'}]
2014-01-03 22:59:00+00:00    [{'id': '3', 'val': 'aea'}, {'id': '4', 'val': 'te'}]

I would like to convert this to Dataframe with columns like:

                           id   val
2014-01-01 22:59:00+00:00   1   ab
2014-01-01 22:59:00+00:00   2   abc
2014-01-02 22:59:00+00:00   1   aa
......

Any idea on how to implement that? Thanks

I tried using pandas pd.dataframe method with different parameters.

df = pd.DataFrame(series)

Answer 1

Your sample is a Pandas Series and not a DataFrame. So create a dataframe with two columns, convert every column and rejoin the DataFrame.

df = pd.concat([sample.apply(pd.Series)[column].apply(pd.Series) for column in df.columns])
print(df.head())

Output:

                           id   val
2014-01-01 22:59:00+00:00   1   ab
2014-01-02 22:59:00+00:00   1   aa
2014-01-03 21:59:00+00:00   3   aea
2014-01-01 22:59:00+00:00   2   abc
2014-01-02 22:59:00+00:00   2   ewe
2014-01-03 21:59:00+00:00   4   te

Answer 2

You can use the method explode() (new in Pandas 0.25.0) to expand your table vertically and the method apply(pd.Series) to expand you table horizontally:

series.explode().apply(pd.Series)

Output:

                          id  val
2014-01-01 22:59:00+00:00  1   ab
2014-01-01 22:59:00+00:00  2  abc
2014-01-02 22:59:00+00:00  1   aa
2014-01-02 22:59:00+00:00  2  ewe
2014-01-03 21:59:00+00:00  3  aea
2014-01-03 21:59:00+00:00  4   te