pandas pivot table，通过获取多列的差异来创建表

Question

I have this pivot table that has a 2 level column filter.

table_pivot = pandas.pivot_table(table_raw, values='PRICE', index=['DATE', 'HOUR'],
                             columns=['TYPE', 'ID'], aggfunc= numpy.mean, fill_value= 0)

output pivot looks like this:

TYPE                 type X         type Y
ID                  X1  X2  X3     Y1  Y2  Y3  Y4
DATE      HOUR 
1/1/2015  1       10  20  30     20  40  60  80
1/1/2015  2       20  40  60     10  50  70  90 

Now I'm trying to get the difference between the types such that intended output looks something like this:

                  Z
                  Y1 - X1    Y1 - X2    Y1 - X3    Y1 - X1   Y2 - X1 ....
Date      Hour 
1/1/2015  1         10         20         30          40        30   ... 
1/1/2015  2        -10         30         50          70       -30   ...

I thought it would be something like:

table_pivot['Z'] = table_pivot['Y'] - table['X']

But it appears It does not work. How do I create a new table to get the difference between all possible XY combinations?

UPDATE: I tried the following lines. However, I've been getting the MemoryError message. Does anyone know how I can remedy this?

x_list = table_pivot['X'].columns.values
y_list = table_pivot['Y'].columns.values

table_diff = pandas.DataFrame()

for each_x in x_list:
    for each_y in y_list:
        colName = each_y + ' - ' + each_x
        table_diff[colName] = table_pivot['Y'][each_y] - table_pivot['X'][each_x]

Answer 1

A case of multi index slicing, sub() and concat.

df = pd.DataFrame({('Y', 'Y4'): {('1/1/2015', 2L): 90, ('1/1/2015', 1L): 80}, ('X', 'X1'): {('1/1/2015', 2L): 20, ('1/1/2015', 1L): 10}, ('X', 'X2'): {('1/1/2015', 2L): 40, ('1/1/2015', 1L): 20}, ('X', 'X3'): {('1/1/2015', 2L): 60, ('1/1/2015', 1L): 30}, ('Y', 'Y3'): {('1/1/2015', 2L): 70, ('1/1/2015', 1L): 60}, ('Y', 'Y1'): {('1/1/2015', 2L): 10, ('1/1/2015', 1L): 20}, ('Y', 'Y2'): {('1/1/2015', 2L): 50, ('1/1/2015', 1L): 40}})
df.columns = pd.MultiIndex.from_tuples([('X','X1'), ('X','X2'), ('X','X3'),('Y','Y1'), ('Y','Y2'), ('Y','Y3'), ('Y', 'Y4')])
df.index.names = ['DATE','ID']
print df


              X           Y            
             X1  X2  X3  Y1  Y2  Y3  Y4
DATE     ID                            
1/1/2015 1   10  20  30  20  40  60  80
         2   20  40  60  10  50  70  90


idx = pd.IndexSlice
collection = []
for tup in filter(lambda x: x[0] == "Y", df.columns.tolist()):
    foo = -1 * df.loc[:,idx['X',:]].sub(df.loc[:,tup],axis=0)
    foo.columns = [str(tup[1]) + '-' + col for col in foo.columns.get_level_values(1)]
    collection.append(foo)

print pd.concat(collection,axis=1)


             Y1-X1  Y1-X2  Y1-X3  Y2-X1  Y2-X2  Y2-X3  Y3-X1  Y3-X2  Y3-X3  Y4-X1  Y4-X2  Y4-X3
DATE     ID                                                                                    
1/1/2015 1      10      0    -10     30     20     10     50     40     30     70     60     50
         2     -10    -30    -50     30     10    -10     50     30     10     70     50     30

Answer 2

Possible better way to do this is by using a time series data frame for each variable, then creating another data frame with the difference between the variables.

data = pd.read_csv('file_path', index_column)

#assuming data is in date-time format
data.index() = pd.to_datetime(data.index())

xvars = data.type['X']
yvars = data.type['Y']

Then use the same for loop logic for taking Yi - Xi and storing that in a new data frame.

May not throw the memory error by keeping objects simple.