应用/向量化/加速逐列清理功能到pandas数据帧

Question

I have some data pipeline code that applies transformation / cleanup logic to columns of a Pandas dataframe based on their name.

Right now I'm iterating over the columns using df.iteritems() which according to this guide on optimizing Pandas apply functions is better than crude looping but is "the least efficient way to run most standard functions".

I'd like to improve the performance of this code either by taking advantage of the Pandas's ability to vectorize these operations, or some other parallel approach.

All of the worked examples I have seen illustrate how to do this row-wise (eg, compute on a Series instead of computing on a single row) but I haven't been able to find a good example of how to do this column-wise .

Here is a reproducible / toy example using the Boston dataset from scikit learn. Desired outcome is to implement the cleaning logic in a vectorized / parallel manner (without using .iteritems() or looping). Thanks!

from typing import Callable

# sample df from sklearn
from sklearn import datasets
boston = datasets.load_boston()
boston = pd.DataFrame(boston.data, columns=boston.feature_names)
boston.head()

def double_it(col: pd.Series) -> pd.Series:
    return col.multiply(2)

def make_string(col: pd.Series) -> pd.Series:
    return col.astype(str)

def do_nothing(col: pd.Series) -> pd.Series:
    return col

def match_cleaner(col_name: str) -> Callable:
    if col_name in ['ZN', 'NOX', 'INDUS', 'AGE']:
        return double_it
    elif col_name in ['TAX', 'DIS', 'CHAS', 'PTRATIO']:
        return make_string
    else:
        print(col_name)
        return do_nothing

for key, value in boston.iteritems():
    cleaning_func = match_cleaner(key)
    boston.loc[:, key] = cleaning_func(value)

# confirm changes
boston.head()
print(boston.dtypes)

Answer 1

You could use pandas.DataFrame.apply . The apply method will by default apply the provided function across all columns in the dataframe. But you would need to modify your match_cleaner function a bit.

def match_cleaner2(col):
     col_name = col.name
     if col_name in ['ZN', 'NOX', 'INDUS', 'AGE']:
         return double_it(col)
     elif col_name in ['TAX', 'DIS', 'CHAS', 'PTRATIO']:
         return make_string(col)
     else:
         return do_nothing(col)

b2 = boston.apply(match_cleaner2)
b2.head()
      CRIM             ZN          INDUS  ...   PTRATIO       B  LSTAT
0  0.00632  3.932955e+246  5.047292e+245  ...      15.3  396.90   4.98
1  0.02731   0.000000e+00  1.544777e+246  ...      17.8  396.90   9.14
2  0.02729   0.000000e+00  1.544777e+246  ...      17.8  392.83   4.03
3  0.03237   0.000000e+00  4.763245e+245  ...      18.7  394.63   2.94
4  0.06905   0.000000e+00  4.763245e+245  ...      18.7  396.90   5.33

%timeit boston.apply(match_cleaner2)
3.68 ms ± 68.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

def original():
     for k, v in boston.iteritems():
         clean_f = match_cleaner(k)
         boston.loc[:, k] = clean_f(v)

original()
boston.head()
      CRIM             ZN          INDUS  ...   PTRATIO       B  LSTAT
0  0.00632  3.932955e+246  5.047292e+245  ...      15.3  396.90   4.98
1  0.02731   0.000000e+00  1.544777e+246  ...      17.8  396.90   9.14
2  0.02729   0.000000e+00  1.544777e+246  ...      17.8  392.83   4.03
3  0.03237   0.000000e+00  4.763245e+245  ...      18.7  394.63   2.94
4  0.06905   0.000000e+00  4.763245e+245  ...      18.7  396.90   5.33


pd.testing.assert_frame_equal(b2, boston) # boston was modified in place

# No AssertionError means frames are equal

%timeit original()
6.14 ms ± 278 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

So from a very rough experiment the apply function looks to speed this up ~40%.