提高熊猫布尔索引的速度

Question

Using the boolean indexing with a sample data worked fine, but as I increased the size of the data, the computing time is getting exponentially long (example below). Anyone knows a way to increase the speed of that particular boolean indexer ?

import pandas as pd
import numpy as np
a = pd.date_range('2019-01-01', '2019-12-31',freq = '1T')
b = np.random.normal(size = len(a), loc = 50)
c = pd.DataFrame(index = a, data = b, columns = ['price'])

1500 rows:

z = c.head(1500)
z[z.index.map(lambda x : 8 <= x.hour <= 16 ) & z.index.map(lambda x : x.weekday() < 5 )]

CPU times: user 149 ms, sys: 8.71 ms, total: 158 ms Wall time: 157 ms

5000 rows:

z = c.head(5000)
z[z.index.map(lambda x : 8 <= x.hour <= 16 ) & z.index.map(lambda x : x.weekday() < 5 )]

CPU times: user 14.1 s, sys: 9.07 s, total: 23.2 s Wall time: 23.2 s

I tried with z = c.head(10000) but it's taking more than 15 minutes to comput so i stopped... The size of the data I want to use that indexer on is about 30000 rows.

Answer 1

Both z.index.map(lambda x : 8 <= x.hour <= 16) and z.index.map(lambda x: x.weekday() < 5) execute almost instantly. The problem occurs when you combine these with the bitwise and operator, & .

pd.Index.map returns another pd.Index object. And the & operator on Index objects actually does set intersection; it is not "element-wise and". If you take a look at the result you will see that it is not what you expect -- it is 5000 True s. The reason it is taking so long is that these comparisons return boolean values which are of course duplicated and index intersection fails in that situation.

The proper way of handling this is of course using vectorized operations but if you somehow need to element-wise compare two pd.Index object, you can do so by converting them to numpy arrays:

res1 = z.index.map(lambda x : 8 <= x.hour <= 16 ).to_numpy()
res2 = z.index.map(lambda x : x.weekday() < 5 ).to_numpy()
z[res1 & res2]

Answer 2

The reason this does not work fast is because you perform a mapping with a lambda expression, so that means that for each item, a function call will be made. This is typically not a good idea if you want to process data in "bulk". You can speed this up with:

hour = z.index.hour
z[]

With z = c (so a total of 524'161 rows), we get the following timings:

>>> z = c
>>> timeit(lambda: z[(8 <= z.index.hour) & (z.index.hour <= 16) & (z.index.weekday < 5)], number=100)
11.825318349001464

So this runs in a total of ~118 milliseconds per run.

When we use the first 5'000 rows, we get:

>>> z = c.head(5000)
>>> timeit(lambda: z[(8 <= z.index.hour) & (z.index.hour <= 16) & (z.index.weekday < 5)], number=100)
0.1542488380218856

So this runs in 1.5 milliseconds per run.