Dictionary in Pandas DataFrame, how to split the columns

Question

I have a DataFrame that consists of one column ('Vals') which is a dictionary . The DataFrame looks more or less like this:

In[215]: fff
Out[213]: 
                                                Vals
0    {u'TradeId': u'JP32767', u'TradeSourceNam...
1    {u'TradeId': u'UUJ2X16', u'TradeSourceNam...
2    {u'TradeId': u'JJ35A12', u'TradeSourceNam...

When looking at an individual row the dictionary looks like this:

In[220]: fff['Vals'][100]
Out[218]: 
{u'BrdsTraderBookCode': u'dffH',
 u'Measures': [{u'AssetName': u'Ie0',
   u'DefinitionId': u'6dbb',
   u'MeasureValues': [{u'Amount': -18.64}],
   u'ReportingCurrency': u'USD',
   u'ValuationId': u'669bb'}],
 u'SnapshotId': 12739,
 u'TradeId': u'17304M',
 u'TradeLegId': u'31827',
 u'TradeSourceName': u'xxxeee',
 u'TradeVersion': 1}

How can I split the the columns and create a new DataFrame , so that I get one column with TradeId and another one with MeasureValues ?

Answer 1

Here's a way to get TradeId and MeasureValues (using twice your sample row above to illustrate the iteration):

new_df = pd.DataFrame()
for id, data in fff.iterrows():
    d = {'TradeId': data.ix[0]['TradeId']}
    d.update(data.ix[0]['Measures'][0]['MeasureValues'][0])
    new_df = pd.concat([new_df, pd.DataFrame.from_dict(d, orient='index').T])

  Amount TradeId
0 -18.64  17304M
0 -18.64  17304M

Answer 2

try this:

l=[]

for idx, row in df['Vals'].iteritems():
    temp_df = pd.DataFrame(row['Measures'][0]['MeasureValues'])
    temp_df['TradeId'] =  row['TradeId']
    l.append(temp_df)

pd.concat(l,axis=0)