dask distributed code is slower than corresponding serial execution

Question

I have this dask example of a standalone python script that runs on my desktop that has 4 CPU nodes It takes 0.735 seconds currently. The goal is to use separate processes on my Linux to overcome the limitations of the GIL etc.

import numpy as np
import dask
from dask import delayed
from dask.distributed import Client
import time

def main():

    if __name__ == "__main__":

      client = Client()
      tmp,pres = setUpData()
      startTime = time.time()
      executeCalc(tmp,pres)
      stopTime = time.time()
      print(stopTime-startTime)

def setUpData():

    temperature = 273 + 20 * np.random.random([4,17,73,144])
    pres = ["1000","925","850","700","600","500","400","300","250","200","150","100","70","50","30","20","10"]
    level = np.array(pres)
    level = level.astype(float)*100
   return temperature,level

 def executeCalc(tmp,pres):
     potempList = []
     for i in range (0,tmp.shape[0]):
        tmpInstant = tmp[i,:,:,:]
        potempList.append(delayed(pot)(tmpInstant,pres))
     results = dask.compute(potempList,scheduler='processes',num_workers=4)
                      
def pot(tmp,pres):
   potemp = np.zeros((17,73,144))
   potemp = tmp * (100000./pres[:,None,None])
   return potemp

main()

Here is the corresponding serial execution with only trivial modifications and this takes 0.0024 seconds.

def executeCalc(tmp,pres):
    potempList = []
    for i in range (0,tmp.shape[0]):
       tmpInstant = tmp[i,:,:,:]
       potemp = pot(tmpInstant,pres)

Where am I going wrong? At the very least for this trivial amount of data the execution times should be identical.

Answer 1

This assumption is where you're going wrong:

At the very least for this trivial amount of data the execution times should be identical.

Any parallel execution engine does the same amount of work as the serial engine, plus a lot of additional overhead. So you need to have enough parallelizeable work to do to justify the large time required to spin up the scheduler, web server, worker processes, transmit the job and results across workers, serialize and deserialize all inputs all outputs, monitor and manage job and worker state, etc.

To see the benefits of dask, you need a task which is many orders of magnitude larger than this.

Take a look at the dask docs on best practices. I'm paraphrasing, butthe first recommendation is don't use dask unless you have to.