熊猫groupby对象唯一计数性能

Question

I have a large dataset of transaction data which looks like:

| cust_no | acct_no | trans_id | product_id | ..... |

I try several way to count how many unique accounts per customer, and how many unique products customer buy etc.

• Method 1.a

transaction_df[['cust_no','acct_no']].groupby('cust_no')['acct_no'].nunique()

which runs average 91.5ms

• Method 1.b

transaction_df.groupby('cust_no')['acct_no'].nunique()

which runs average 85.5ms

• Method 2.a

transaction_df[['cust_no','acct_no']].groupby(['cust_no','acct_no']).size().groupby('cust_no').size()

which runs 61.5ms

• Method 2.a

transaction_df.groupby(['cust_no','acct_no']).size().groupby('cust_no').size()

which runs 55.3ms

I have two question:

1. why is the the DataFrame after slicing run slower, ie transaction_df[['cust_no','acct_no']] is slower than just transaction_df ?

2. why .nunique() method is much slower than just stack up two groupby ?

Answer 1

1) Slicing requires memory assignment and/or a copy of the object depending on the operation. Here you're creating a new DataFrame before starting your operations.

2) nunique is going to either implement logic for or directly call a set , which runs in O(N) time. size will run O(1)

Knowing prior structural information about your dataset can help you optimize function selection as you're experimenting with here. Read into https://en.wikipedia.org/wiki/Time_complexity if you're interested