How do you create a pandas DataFrame from a list with multiple dictionaries with nested values?

Question

Pretty new to Python still and I'm trying to figure out how to create a properly formatted DataFrame from a list of dictionaries I created.

listOutput = 
[{0: ['Name', val1, val2, val3, val4, val5]}, 
{1: ['Name', val1, val2, val3, val4, val5]}]

Into something like:

      0               1 
0  Name1            Name2
1  val1             val1
2  val2             val2
3  val3             val3
4  val4             val4
5  val5             val5

When I only make a DataFrame from one list only, it's properly formatted but when I make one from the list with Dictionaries, it outputs something like this:

 0                                                  1
0  [Name1, 7995, 138.5, 300.0, 50.0, 7506.5] NaN                                   
1  NaN                                              [Name2, 7995,138.5, 300.0, 50.0, 75...

Answer 1

Use a dictionary comprehension to merge the dictionaries:

import pandas as pd
df = pd.DataFrame({k:v for d in listOutput for k,v in d.items()})

Alternative using collections.ChainMap (a bit slower):

from collections import ChainMap
import pandas as pd

df = pd.DataFrame(dict(ChainMap(*listOutput)))

Output:

      0     1
0  Name  Name
1  val1  val1
2  val2  val2
3  val3  val3
4  val4  val4
5  val5  val5

Answer 2

Since each dictionary in the list represents a column with its header, you could usepd.concat along axis=1

pd.concat([pd.DataFrame(x) for x in listOutput], axis=1)

Explanation: 2 parts: Creating DataFrame in list comprehension + pd.concat()

1. In the list comprehension you iterate through each element of your input list listOutput . Each element in this list is a dictionary with a key and a list as value. When creating a DataFrame you can use exactly that where key -> column name, value -> column data. Considering your list looks like this:

listOutput = [{0: ['Name', 'val1', 'val2', 'val3']}, 
              {1: ['Name', 'val4', 'val5', 'val6']}]

your two dfs created during iteration look like this:

#first iteration (e.g df1):
pd.DataFrame({0: ['Name', 'val1', 'val2', 'val3']})

      0
0  Name
1  val1
2  val2
3  val3

# second iteration (e.g df2):
pd.DataFrame({1: ['Name', 'val4', 'val5', 'val6']})

      1
0  Name
1  val4
2  val5
3  val6

2. These 2 dfs are stored in a list and passed to pd.concat . axis=1 means concatenation along the columns. It expects to get a "a sequence or mapping of Series or DataFrame objects" (->documentation ), here you have a sequence of DataFrame objects. We didn't assign the dfs during the loop to a variable (because we don't need to), but considered you would name them (like I did above in the parenthesis), then in your last step concatenating the dfs would look like this:

pd.concat([df1, df2], axis=1)

      0     1
0  Name  Name
1  val1  val4
2  val2  val5
3  val3  val6

Answer 3

This format is a bit scuffed if we want it to look good in pandas.

listOutput = [{0: ['Name1', 1, 2, 3, 4, 5]},
              {1: ['Name2', 6, 7, 8, 9, 10]}]

---

If you have control over this list, you can re-format it like this:

listOutput = {'Name1': [1, 2, 3, 4, 5],
              'Name2': [6, 7, 8, 9, 10]}

Which leads to this:

>>> pd.DataFrame(listOutput)
   Name1  Name2
0      1      6
1      2      7
2      3      8
3      4      9
4      5     10

---

If you don't have control, you can fix it like this:

# This extracts the values from each dictionary in your list, and makes it
# into a properly formatted dictionary.
listOutput = {x[0]:x[1:] for x in [list(y.values())[0] for y in listOutput]}
# Produces same output as above~

Answer 4

Another possible solution, based onpandas.Series and on pandas.concat :

pd.concat(
  pd.Series(listOutput)
  .map(lambda x: pd.DataFrame.from_dict(x)).to_list(), axis = 1)

Output:

      0     1
0  Name  Name
1     1     1
2     2     2
3     3     3
4     4     4
5     5     5