pandas DF out of list of dictionaries with index = keys

Question

I want to pass a list of dicts into a dataframe:

mylist_of_dicts= [{'H1': 2},
                  {'H2': 2},
                  {'H3': 3},
                  {'H4': 2},
                  {'H5': 2},
                  {'B6': 2},
                  {'H7': 2},
                  {'H8': 2}]
cl_ocurrences = pd.DataFrame(mylist)
print(cl_ocurrences.head(2))
cl_ocurrences = pd.DataFrame.from_dict(mylist)
print(cl_ocurrences.head(2))
cl_ocurrences = pd.DataFrame.from_records(mylist)
print(cl_ocurrences.head(2))

But I would like that the keys appear in a column and the values in other column.

I could play around with list comprehensions and pass two lists (values and keys) to the DF constructor, but there should be a nicer way.

Answer 1

I'm not sure that there is an explicit function in Pandas to unravel a list of dicts in the format that you want.

This uses a combination of reduce and the new union method for dictionaries ( I think it comes into effect in python 3.9):

from functools import reduce
merged_dict = reduce(lambda x, y: x|y, mylist)
(pd.DataFrame.from_dict(merged_dict, 
                       orient = 'index', 
                       columns = ['value'])
 .rename_axis('H')
 .reset_index()
 )

    H  value
0  H1      2
1  H2      2
2  H3      3
3  H4      2
4  H5      2
5  B6      2
6  H7      2
7  H8      2

For python versions less than 3.9, unpacking works fine too:

merged_dict = reduce(lambda x, y: {**x, **y}, mylist)

You could also use a combination of itertools and map :

pd.DataFrame(chain.from_iterable(map(lambda x: x.items(), mylist)), 
             columns = ['H', 'value'])

Answer 2

If you have a fixed structure of your individual dicts (eg {'H':value} for all of them, then I would redefine the individual dicts such that they have the same labels:

extended_list = [{'name': next(iter(x)), 'value': x[next(iter(x))]} for x in mylist]
cl_ocurrences = pd.DataFrame.from_dict(extended_list)
print(cl_ocurrences.head())

which results in

  name  value
0   H1      2
1   H2      2
2   H3      3
3   H4      2
4   H5      2