熊猫中基于两个条件的分组和聚集的最有效方法

Question

I have the following dataframe :

user_id  sale     date        refunded
  1      1000    '2016-10-02'   0  
  1      1000    '2016-09-13'   0
  2      1000    '2016-08-11'   0
  2      1000    '2016-10-21'   0
  3      1000    '2016-11-01'   1
  3      1000    '2016-11-01'   1

i need to group by user_id and calculate sum of sale based on these two conditions:

   date >='2016-10-01'
   refunded==0

i took two different approach:

    non_refunded = df.refunded == 0
    after_assignment = df.date > '2016-10-01'
    columns = ['user_id', 'sale']
    tt = tdf.loc[non_refunded & after_assignment][columns].groupby(['user_id']).sum().reset_index()

another approach is:

columns = ['user_id', 'sale']
tt = df.loc[(df.refunded == 0) & (df.date > '2016-10-01')][columns].groupby(['user_id']).sum().reset_index()

in first approach first i create two copy dataframe (i am not sure if they are copy) then apply the condition.how do you compare these two approaches in terms of speed, resources needed, when these two approaches start to show their differences , for example if we should do it for 30 different dataframes with 100k rows or more.

Answer 1

Are you using an IPython interpreter? If so, you can use the %timeit magic to measure how long it takes to execute one line of code. You two approaches seem to do the exact same thing - I wouldn't expect any performance difference.

For readability I would use the second approach:

%timeit df.loc[(df.refunded == 0) & (df.date > '2016-10-01')].groupby('user_id').sum()

Pandas won't struggle with 100k row data frames on an reasonably modern laptop.

Answer 2

I think you can use query :

df.date = pd.to_datetime(df.date)
columns = ['user_id', 'sale']
filtered = df.query('refunded == 0 and date > "2016-10-01"')
tt = filtered[columns].groupby(['user_id']).sum().reset_index()
print (tt)
   user_id  sale
0        1  1000
1        2  1000

Another solution is removed ][ and add columns to loc :

df.date = pd.to_datetime(df.date)
columns = ['user_id', 'sale']
filtered = df.loc[(df.refunded == 0) & (df.date > '2016-10-01'), columns]
tt = filtered[columns].groupby(['user_id']).sum().reset_index()
print (tt)
   user_id  sale
0        1  1000
1        2  1000