加快将函数应用于pandas数据帧列表

Question

I have some time series data where each data point is a pandas data frame ie list of data frames. I also have a function foo which operates on each data point. The aim is to apply the function over the entire time series data and do it efficiently.

I have vectorized the function foo so that it operates on the entire data frame and achieved a speedup of around 32x.

The original code is as follows:

def bar(row, cols):
    return tuple([row[col] for col in cols])

def foo(df, cols):
    keys = set()
    for index, row in df.iterrows():
        key = bar(row, cols)
        keys.add(key)

    # do calculations on keys that returns a numeric output, result
    return result # float64

The vectorized code is as follows:

def vect_bar(df, cols):
    df['key'] = df[cols].values.sum(axis=1)
    return df

def vect_foo(df, cols):
    df['key'] = ""
    df = vect_bar(df, cols)
    keys = df.key.unique()

    # do calculations on keys that returns a numeric output, result
    return result # float64

The timing results are as follows:

%timeit -n 100 foo(df, cols)
100 loops, best of 3: 42.9 ms per loop

%timeit -n 100 vect_foo(df, cols)
100 loops, best of 3: 1.34 ms per loop

Note: cols is a list of column names. All the elements of the data frame are strings and of dtype object.

However, it still takes a long time to apply vect_foo to all the data points. How can I speed it up further?

I tried creating a pandas series from the data and using series.apply() . However, that did not cause any speedup from the regular for loop approach.

EDIT : If I was not clear earlier, given a data frame, the function vect_foo is quite efficient. What I want is a way to speed up applying vect_foo to all the data points ie the list of data frames.

data_series = pd.Series(data)
def apply_data():
    return data_series.apply(vect_foo, cols)

data is a list of pandas data frames ie data = [df1, df2, ..., df50K]

Here, I tried pandas.Series.apply() but it performed similarly to a normal for loop approach.

Answer 1

import pandas as pd

def foo(row, cols):
    row['keys'] = row[cols].sum()
    return row

df.apply(foo, axis=1)

just create your helper function and use the apply function. this is usually the most efficient way to apply a function across rows/columns in pandas