Convert list of identical dictionaries to Dataframe

Question

I have a list like this:

[{'FirstOfficer': '1'}, {'SecondOfficer': '2'}, {'ThirdOfficer': '3'},{'FirstOfficer': '4'}, {'SecondOfficer': '5'}, {'ThirdOfficer': '6'},{'FirstOfficer': '7'}, {'SecondOfficer': '8'}, {'ThirdOfficer': '9'},{'FirstOfficer': '10'}, {'SecondOfficer': '11'}, {'ThirdOfficer': '12'}]

I wanted to convert this into a dataframe but i got the dataframe like this:

   FirstOfficer SecondOfficer ThirdOfficer
0             1           NaN          NaN
1           NaN             2          NaN
2           NaN           NaN            3
3             4           NaN          NaN
4           NaN             5          NaN
5           NaN           NaN            6
6             7           NaN          NaN
7           NaN             8          NaN
8           NaN           NaN            9
9            10           NaN          NaN
10          NaN            11          NaN
11          NaN           NaN           12

the columns name can be anything, so I am not able to hard code it.

Expected dataframe is:

   FirstOfficer SecondOfficer ThirdOfficer
0             1           2          3
1             4           5          6
2             7           8          9
3            10          11         12

Can anybody suggest me a solution for it?

Any help would be appreciated.

Answer 1

Use defaultdict for store values to list by keys of dictionaries:

from collections import defaultdict

d = defaultdict(list)
for x in L:
    a, b = tuple(x.items())[0]
    d[a].append(b)
print (d)


df = pd.DataFrame(d)
print (df)
  FirstOfficer SecondOfficer ThirdOfficer
0            1             2            3
1            4             5            6
2            7             8            9
3           10            11           12

Answer 2

If performance is not an issue, you can use:

df=pd.DataFrame(l).apply(lambda x: pd.Series(x.dropna().values))
print(df)

---

  FirstOfficer SecondOfficer ThirdOfficer
0            1             2            3
1            4             5            6
2            7             8            9
3           10            11           12

Answer 3

d = [{'FirstOfficer': '1'}, {'SecondOfficer': '2'}, {'ThirdOfficer': '3'}, {'FirstOfficer': '4'}, {'SecondOfficer': '5'}, {'ThirdOfficer': '6'}, {'FirstOfficer': '7'}, {'SecondOfficer': '8'}, {'ThirdOfficer': '9'}, {'FirstOfficer': '10'}, {'SecondOfficer': '11'}, {'ThirdOfficer': '12'}]

keys = list(set([str(i.keys()).split("'")[1] for i in d]))
final_dict = dict()
for key in keys:
    final_dict['key'] = [i[key] for i in d if key in i.keys()]
df = pd.DataFrame.from_dict(final_dict)

OUTPUT:

  FirstOfficer SecondOfficer ThirdOfficer
0            1             2            3
1            4             5            6
2            7             8            9
3           10            11           12

Answer 4

One approach is to pre-process your list

Ex:

import pandas as pd

lst = [{'FirstOfficer': '1'}, {'SecondOfficer': '2'}, {'ThirdOfficer': '3'},{'FirstOfficer': '4'}, {'SecondOfficer': '5'}, {'ThirdOfficer': '6'},{'FirstOfficer': '7'}, {'SecondOfficer': '8'}, {'ThirdOfficer': '9'},{'FirstOfficer': '10'}, {'SecondOfficer': '11'}, {'ThirdOfficer': '12'}]

data = []
for i in range(0, len(lst), 3):
    temp = []
    for d in lst[i:i+3]:
        for _, v in d.items():
            temp.append(v)
    data.append(temp)

df = pd.DataFrame(data, columns=["FirstOfficer", "SecondOfficer", "ThirdOfficer"]) 
print(df)

Output:

  FirstOfficer SecondOfficer ThirdOfficer
0            1             2            3
1            4             5            6
2            7             8            9
3           10            11           12