pandas: stacking DataFrames generated by apply

Question

With a DataFrame , you can output Series when using DataFrame.apply to generate a new DataFrame with new columns

          a         b         c
0 -0.119342  0.286710  0.266750
1 -1.514301  0.556106 -2.743888
2 -0.156469 -0.352915 -1.963398
3  1.165479  1.364303  0.648178
4  1.541738  0.714239 -1.468896

def f(x):
    return pandas.Series([ x['a']+x['b'], x['b'] + x['c'], x['a'] + x['c'] ], index=['ab', 'bc', 'ac'])

In [52]: df.apply(f, axis=1)
Out[52]: 
         ab        bc        ac
0  0.167368  0.553460  0.147408
1 -0.958195 -2.187782 -4.258188
2 -0.509384 -2.316313 -2.119867
3  2.529782  2.012481  1.813658
4  2.255977 -0.754657  0.072842

Attempting to output new DataFrame objects instead of Series objects results in stacking objects rather than creating a cohesive DataFrame

In [53]: def f(x):
    return pandas.DataFrame([[ x['a']+x['b'], x['b'] + x['c'], x['a'] + x['c'] ]], columns=['ab', 'bc', 'ac'])
   ....: 

In [54]: df.apply(f, axis=1)
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Out[54]: 
0             ab       bc        ac
0  0.167368  0.553
1             ab        bc        ac
0 -0.958195 -2.18
2             ab        bc        ac
0 -0.509384 -2.31
3             ab        bc        ac
0  2.529782  2.01
4             ab        bc        ac
0  2.255977 -0.75

Is there a way to output DataFrame s (or multiple Series ) that can be stacked the way outputting single Series objects are?

Answer 1

What you are trying to do seem to work fine, except for the warnings in the repr (for which i created an issue: https://github.com/pydata/pandas/issues/1749 )

In [57]: df
Out[57]:
          a         b         c
0 -0.119342  0.286710  0.266750
1 -1.514301  0.556106 -2.743888
2 -0.156469 -0.352915 -1.963398
3  1.165479  1.364303  0.648178
4  1.541738  0.714239 -1.468896

In [58]: s = df.apply(f, axis=1)

In [59]: type(s)
Out[59]: pandas.core.series.Series

In [60]: type(s[0])
Out[60]: pandas.core.frame.DataFrame

In [61]: s[0]
Out[61]:
         ab       bc        ac
0  0.167368  0.55346  0.147408

In [62]: s
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Exception ValueError: ValueError('Cannot call bool() on DataFrame.',) in 'util._checknull' ignored
Out[62]:
0             ab       bc        ac
0  0.167368  0.553
1             ab        bc        ac
0 -0.958195 -2.18
2             ab        bc        ac
0 -0.509384 -2.31
3             ab        bc        ac
0  2.529782  2.01
4             ab        bc        ac
0  2.255977 -0.75