Fastest way to query a dataframe

Question

I want to make aggregation operations (sum) on the rows of a big pandas dataframe(millions of rows) which are determined by a condition on several fixed columns (max 10 columns). These columns have only integer values.

My problem is that I have to make this operation (querying + aggregating) thousands of times (~100 000 times). I think with the aggregating part there is not much to optimize as it is just a simple sum. What would be the most efficient way to perform this task? Is there some way I could build an 'index' on my condition columns in order to speed up each query?

Answer 1

I would try something in this flavor:

Suppose you have the following dataframe

N = 10000000
df = pd.DataFrame({
    'A':np.random.binomial(1,0.5,N),
    'B':np.random.binomial(2,0.5,N),
    'nume1':np.random.uniform(0,1,N),
    'nume2':np.random.normal(0,1,N)})

then doing this

tmp = df[['A','B','nume1','nume2']].query('A > 0.5').groupby('B').sum().reset_index()[['B','nume1','nume2']]

is the SQL equivalent of

select B, sum(nume1),sum(nume2)
from df
where A > 0.5
group by B

this takes a little less then a sec (926ms, using %timeit) on my moderate (i7 quad-core, 16GB ram) machine.

I hope this helps.

Answer 2

Without more details it's hard to answer your question.

You should indeed build an index of your conditional columns.

df['idx'] = (df['col1'] * df['col2']) ** (df['col3'] + df['col4']) * df['col5'] == 0.012
df = df.set_index('idx')

Rewriting your condition to an indexable column may be hard. Keep in mind you can set all the columns as the index

df = df.set_index(['col1', 'col2', 'col3', 'col4', 'col5' ...])

This documentation on advanced indexing in Pandas may help you think about your problem: http://pandas.pydata.org/pandas-docs/stable/indexing.html#multiindex-query-syntax