Dask Grouby Performance for nunique is too slow. How to improve the performance?

Question

I have large files of size more than 5GB. I have stored them in parquet format. When I do groupby operation as shown below code for small sample set of 600k+ records, Dask is taking more than 6 mins, whereas pandas took only 0.4 seconds. Though I understand pandas is faster if the dataset fits in memory, my question is if I pass entire parquet file to Dask dataframe, will performance improve?

Also suggest me how to improve the below code, so that i can run in few seconds rather than in minutes.

Example: Using Dask Dataframe

StartTime = datetime.datetime.now()
df = dd.read_parquet('201908.parquet', columns=['A', 'B'], engine='pyarrow')
print (len(df))
df = df.set_index ('A')
rs = df.groupby('A').B.nunique().nlargest(10).compute(scheduler='processes')
print (rs)
EndTime = datetime.datetime.now()
print ("Total Time Taken for processing: " + str(EndTime - StartTime))

Output is:

606995
A
-3868378286825833950    7
 1230391617280615928    7
 381683316762598393     6
-5730635895723403090    5
 903278193888929465     5
 2861437302225712286    5
-9057855329515864244    4
-8963355998258854688    4
-7876321060385968364    4
-6825439721748529898    4
Name: B, dtype: int64
Total Time Taken for processing: 0:06:05.042146

Example using Pandas:

StartTime = datetime.datetime.now()
df = pd.read_parquet('201908.parquet', columns=['A', 'B'], engine='pyarrow')
print (len(df))
df = df.set_index ('A')
rs = df.groupby('A').B.nunique().nlargest(10)
print (rs)
EndTime = datetime.datetime.now()
print ("Total Time Taken for processing: " + str(EndTime - StartTime))

Output is:

606995
A
-3868378286825833950    7
 1230391617280615928    7
 381683316762598393     6
-5730635895723403090    5
 903278193888929465     5
 2861437302225712286    5
-9057855329515864244    4
-8963355998258854688    4
-7876321060385968364    4
-6825439721748529898    4
Name: B, dtype: int64
Total Time Taken for processing: 0:00:00.419033

Answer 1

I believe that there is an open issue for an approximate groupby nunique algorithm for dask dataframe. You might look into that if you're particularly interested. Dask dataframe's non-groupby nunique algorithm is quite a bit faster.