为什么列表理解比在 pandas 中应用更快

Question

Using List comprehensions is way faster than a normal for loop. Reason which is given for this is that there is no need of append in list comprehensions, which is understandable. But I have found at various places that list comparisons are faster than apply. I have experienced that as well. But not able to understand as to what is the internal working that makes it much faster than apply?

I know this has something to do with vectorization in numpy which is the base implementation of pandas dataframes. But what causes list comprehensions better than apply, is not quite understandable, since, in list comprehensions, we give for loop inside the list, whereas in apply, we don't even give any for loop (and I assume there also, vectorization takes place)

Edit: adding code: this is working on titanic dataset, where title is extracted from name: https://www.kaggle.com/c/titanic/data

%timeit train['NameTitle'] = train['Name'].apply(lambda x: 'Mrs.' if 'Mrs' in x else \
                                         ('Mr' if 'Mr' in x else ('Miss' if 'Miss' in x else\
                                                ('Master' if 'Master' in x else 'None'))))

%timeit train['NameTitle'] = ['Mrs.' if 'Mrs' in x else 'Mr' if 'Mr' in x else ('Miss' if 'Miss' in x else ('Master' if 'Master' in x else 'None')) for x in train['Name']]

Result: 782 µs ± 6.36 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

499 µs ± 5.76 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Edit2: To add code for SO, was creating a simple code, and surprisingly, for below code, the results reverse:

import pandas as pd
import timeit
df_test = pd.DataFrame()
tlist = []
tlist2 = []
for i in range (0,5000000):
  tlist.append(i)
  tlist2.append(i+5)
df_test['A'] = tlist
df_test['B'] = tlist2

display(df_test.head(5))


%timeit df_test['C'] = df_test['B'].apply(lambda x: x*2 if x%5==0 else x)
display(df_test.head(5))
%timeit df_test['C'] = [ x*2 if x%5==0 else x for x in df_test['B']]

display(df_test.head(5))

1 loop, best of 3: 2.14 s per loop

1 loop, best of 3: 2.24 s per loop

Edit3: As suggested by some, that apply is essentially a for loop, which is not the case as if i run this code with for loop, it almost never ends, i had to stop it after 3-4 mins manually and it never completed during this time.:

for row in df_test.itertuples():
  x = row.B
  if x%5==0:
    df_test.at[row.Index,'B'] = x*2

Running above code takes around 23 seconds, but apply takes only 1.8 seconds. So, what is the difference between these physical loop in itertuples and apply?

Answer 1

There are a few reasons for the performance difference between apply and list comprehension.

First of all, list comprehension in your code doesn't make a function call on each iteration, while apply does. This makes a huge difference:

map_function = lambda x: 'Mrs.' if 'Mrs' in x else \
                 ('Mr' if 'Mr' in x else ('Miss' if 'Miss' in x else \
                 ('Master' if 'Master' in x else 'None')))

%timeit train['NameTitle'] = [map_function(x) for x in train['Name']]
# 581 µs ± 21.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit train['NameTitle'] = ['Mrs.' if 'Mrs' in x else \
                 ('Mr' if 'Mr' in x else ('Miss' if 'Miss' in x else \
                 ('Master' if 'Master' in x else 'None'))) for x in train['Name']]
# 482 µs ± 14.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Secondly, apply does much more than list comprehension. For example it tries to find appropriate dtype for the result. By disabling that behaviour you can see what impact it has:

%timeit train['NameTitle'] = train['Name'].apply(map_function)
# 660 µs ± 2.57 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit train['NameTitle'] = train['Name'].apply(map_function, convert_dtype=False)
# 626 µs ± 4.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

There's also a bunch of other stuff happening within apply , so in this example you would want to use map :

%timeit train['NameTitle'] = train['Name'].map(map_function)
# 545 µs ± 4.02 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Which performs better than list comprehension with a function call in it.

Then why use apply at all you might ask? I know at least one example where it outperforms everything else -- when the operation you want to apply is a vectorized universal function . That's because apply unlike map and list comprehension allows the function to run on the whole Series instead of individual objects in it. Let's see an example:

%timeit train['AgeExp'] = train['Age'].apply(lambda x: np.exp(x))
# 1.44 ms ± 41.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit train['AgeExp'] = train['Age'].apply(np.exp)
# 256 µs ± 12.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit train['AgeExp'] = train['Age'].map(np.exp)
# 1.01 ms ± 8.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit train['AgeExp'] = [np.exp(x) for x in train['Age']]
# 1.21 ms ± 28.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)