Transform from pandas Series list of dicts to DataFrame

Question

I have a pandas Series of dicts. I would like to convert that to pandas DataFrame. All dicts have 'name' and 'value'. I would like to convert name into column names and values as rows.

I tried to use simply pd.DataFrame(mySeriesOfDicts) or convert Series to list first but non of that works.

from_dicts gives me bad results:

name  value
svSum7Days 0.0 
svSum91Days 0.0 
svSum364Days 423.0 
newPositionsCount60Days 0.0 

and when I try to add orient='index' I get AttributeError: 'list' object has no attribute 'values'

One row of my Series:

[{'name': 'svSum7Days', 'value': 0.0},
 {'name': 'svSum91Days', 'value': 0.0},
 {'name': 'svSum364Days', 'value': 423.0},
 {'name': 'newPositionsCount60Days', 'value': 0.0}]

Series few cutted rows:

0      [{'name': 'svSum7Days', 'value': 0.0}, {'name'...
1      [{'name': 'svSum7Days', 'value': 5.0}, {'name'...
2      [{'name': 'svSum7Days', 'value': 0.0}, {'name'...
3      [{'name': 'svSum7Days', 'value': 0.0}, {'name'...

I would like to have this converted to form:

svSum7Days svSum91Days svSum364Days newPositionsCount60Days
0.0        0.0         423.0        0.0
..         ..          ..           ..

Answer 1

Flatten the inner dictionaries and the construct a dataframe from the list of dictionaries:

d = [{d['name']:d['value'] for d in l} for l in s] # s being the pd.Series
df = pd.DataFrame(d)

---

Quick check -

s = pd.Series([[{'name': 'svSum7Days', 'value': 0.0},
 {'name': 'svSum91Days', 'value': 0.0},
 {'name': 'svSum364Days', 'value': 423.0},
 {'name': 'newPositionsCount60Days', 'value': 0.0}],
          [{'name': 'svSum7Days', 'value': 1.0},
 {'name': 'svSum91Days', 'value': 12.0},
 {'name': 'svSum364Days', 'value': 424.0},
 {'name': 'newPositionsCount60Days', 'value': 100.0}]])

d = [{d['name']:d['value'] for d in l} for l in s]
# [{'svSum7Days': 0.0, 'svSum91Days': 0.0, 'svSum364Days': 423.0, 
#   'newPositionsCount60Days': 0.0}, {'svSum7Days': 1.0, ...
pd.DataFrame(d)

    newPositionsCount60Days  svSum364Days  svSum7Days  svSum91Days
0                      0.0         423.0         0.0          0.0
1                    100.0         424.0         1.0         12.0

Answer 2

Dataframe can be created from a list of dictionaries. But based on your output, you are looking for the transformed data frame.

#your code goes here

import pandas as pd

#creating dummy series
data = pd.Series([{'name': 'svSum7Days', 'value': 0.0},
 {'name': 'svSum91Days', 'value': 0.0},
 {'name': 'svSum364Days', 'value': 423.0},
 {'name': 'newPositionsCount60Days', 'value': 0.0}])


# Convert Series to list
data = data.tolist()

# Create a dataframe
df = pd.DataFrame(data)

# Transpose it
df = df.T

# Create column names
headers = df.iloc[0]
new_df  = pd.DataFrame(df.values[1:], columns=list(headers))
print(new_df)