在没有for循环的情况下将计算出的列添加到Panel内的每个DataFrame中

Question

I have ~300 .csv files all with the same number of rows and columns for instrumentation data. Since each .csv file represents a day and the structure is the same, I figured it would be best to pull each .csv into a Pandas DataFrame and then throw them into a Panel object to perform faster calculations.

I would like to add additional calculated columns to each DataFrame that is inside the Panel, preferably without a for-loop. I'm attempting to use the apply function to the panel and name the new columns based on the original column name appended with a 'p' (for easier indexing later). Below is the code I am currently using.

import pandas as pd
import numpy as np
import os.path

dir = "data/testsetup1/"
filelist = []

def initializeDataFrames():
    for f in os.listdir(dir):
        if ".csv" in f:
                filelist.append(dir + f)

    dd={}
    for f in filelist:
        dd[f[len(dir):(len(f)-4)]] = pd.read_csv(f)

    return pd.Panel(dd)

def newCalculation(pointSeries):
#test function, more complex functions to follow

    pointSeriesManiuplated = pointSeries.copy()

    percentageMove = 1.0/float(len(pointSeriesManiuplated)) 

    return pointSeriesManiuplated * percentageMove


myPanel = initializeDataFrames()
#calculatedPanel = myPanel.join(lambda x: myPanel[x,:,0:17].apply(lambda y:newCalculation(myPanel[x,:,0:17].ix[y])), rsuffix='p')
calculatedPanel = myPanel.ix[:,:,0:17].join(myPanel.ix[:,:,0:17].apply(lambda x: newCalculation(x), axis=2), rsuffix='p')

print calculatedPanel.values

The code above currently duplicates each DataFrame with the calculated columns instead of appending them to each DataFrame. The apply function I'm using operates on a Series object, which in this case would be a passed column. My question is how can I use the apply function on a Panel such that it calculates new columns and appends them to each DataFrame?

Thanks in advance.

Answer 1

If you want to add a new column via apply simply assign the output of the apply operation to the column you desire:

myPanel['new_column_suffix_p'] = myPanel.apply(newCalculation)

If you want multiple columns you can make a custom function for this:

def calc_new_columns (rowset):
    rowset['newcolumn1'] = calculation1(rowset.columnofinterest)
    rowset['newcolumn2'] = calculation2(rowset.columnofinterest2 + rowset.column3)
    return rowset
myPanel = myPanel.apply(calc_new_columns)

On a broader note. You are manually handling sections of your data frame when it looks like you can just do the new column operation all at once. I would suggest importing the first csv file into a data frame. Then loop through the remaining 299 csv and use DataFrame.append to add to the original data frame. Then you would have one data frame for all the data that simple needs the calculated column added.

Answer 2

nit: "dir" is a builtin function. you shouldn't use it as a variable name.

Try using a double transpose:

p = pd.Panel(np.random.rand(4,10,17),
             items=pd.date_range('2013/11/10',periods=4),
             major_axis=range(10),
             minor_axis=map(lambda x: "col%d" % x, range(17)))

pT = p.transpose(2,1,0)
pT = pT.join(pT.apply(newCalculation, axis='major'), rsuffix='p')
p = pT.transpose(2,1,0)