熊猫数据框字典列表

Question

I have a list of dictionaries that I want to convert to a dataframe. Here's what I'm doing:

comments = getComments(submission) #returns list of dicts
tree = flattenTree(comments) #this just removes indentation from one of the text fields
df = pd.DataFrame(tree)['data']

df.head() returns:

0    {u'subreddit_id': u't5_2qj9g', u'banned_by': N...
1    {u'subreddit_id': u't5_2qj9g', u'banned_by': N...
2    {u'subreddit_id': u't5_2qj9g', u'banned_by': N...
3    {u'subreddit_id': u't5_2qj9g', u'banned_by': N...
4    {u'subreddit_id': u't5_2qj9g', u'banned_by': N...
Name: data, dtype: object

raw data is a list of nested dictionaries:

[{u'data': {u'approved_by': None,
u'archived': False,
u'author': u'des-tal',
u'controversiality': 0,
...
u'user_reports': []},
u'kind': u't1'},
{u'data': {u'approved_by': None,
u'archived': False,
...

The format I'm looking for is:

which I can get by selecting rows from the dataframe like this:

...
df = pd.DataFrame(tree)['data']
inddf = pd.DataFrame([df[0],df[1],df[3]])
print inddf

How can I form my dataframe from my dataset for all rows without manually selecting all the rows? I was trying to iterate through the index, but I'm sure there's a better way.

Thanks

Answer 1

you can pass list of dictionaries to pandas dataframe. For example see below

my_list = [

{u'data': {u'approved_by': None,
u'archived': False,
u'author': u'des-tal',
u'controversiality': 0,
u'user_reports': []},
u'kind': u't1'},

 {u'data': {u'approved_by': None,
u'archived': True,
u'author': u'des-tal',
u'controversiality': 0,
u'user_reports': []},
u'kind': u't1'}

]

import pandas as pd
df = pd.DataFrame([i['data'] for i in my_list])
print df.head()

results in

  approved_by archived   author  controversiality user_reports
0        None    False  des-tal                 0           []
1        None     True  des-tal                 0           []

Answer 2

If every dictionary has the same keys, then this should work for what I think you're trying to do.

cols = list_of_dicts[0]['data'].keys()
cols = list(cols)
df=pd.DataFrame(columns=cols)
for d in list_of_dicts:
    df.append(d['data'], ignore_index=True)

If not, make sure you use a representative dictionary to initialize the dataframe. A little slow because it's in a for loop, but should do the trick.