Savgol 过滤 dataframe 列

Question

I'm trying to apply a savgol filter from SciPy to smooth my data. I've successfully applied the filter by selecting each column separately, defining a new y value and plotting it. However I wanted to apply the function in a more efficient way across a dataframe.

y0 = alldata_raw.iloc[:,0]
w0 = savgol_filter(y0, 41, 1)

My first thought was to create an empty array, write a for loop apply the function to each column, append it to the array and finally concatenate the array. However I got an error 'TypeError: cannot concatenate object of type ""; only pd.Series, pd.DataFrame, and pd.Panel (deprecated) objs are valid'

smoothed_array = []
for key,values in alldata_raw.iteritems():
    y = savgol_filter(values, 41, 1)
    smoothed_array.append(y)

alldata_smoothed = pd.concat(smoothed_array, axis=1)

Instead I tried using the pd.apply() function however I'm having issues with that. I have an error message: 'TypeError: expected x and y to have same length'

alldata_smoothed = alldata_raw.apply(savgol_filter(alldata_raw, 41, 1), axis=1)
print(alldata_smoothed)

I'm quite new to python so any advice on how to make each method work and which is preferable would be appreciated!

Answer 1

In order to use the filter first create a function that takes a single argument - the column data. Then you can apply it to dataframe columns like this:

from scipy.signal import savgol_filter
def my_filter(x):
    return savgol_filter(x, 41, 1)
alldata_smoothed = alldata_raw.apply(my_filter)

You could also go with a lambda function:

alldata_smoothed = alldata_raw.apply(lambda x: savgol_filter(x,41,1))

axis=1 in apply is specified to apply the function to dataframe rows. What you need is the default option axis=0 which means apply it to the columns.

That was pretty general but the docs for savgol_filter tell me that it accepts an axis argument too. So in this specific case you could apply the filter to the whole dataframe at once. This will probably be more performant but I haven't checked =).

alldata_smoothed = pd.DataFrame(savgol_filter(alldata_raw, 41, 1, axis=0),
                                columns=alldata_raw.columns,
                                index=alldata_raw.index)