Submit a dask process only if it's not active in any of the workers

Question

Assume I have N workers. I need to submit a program with dask client.submit in one of the workers, but only if it's not running in any of the workers. Any hints on how can this be achieved? Can I get this information from the scheduler?

Answer 1

Dask tokenises your call to a function submitted with client.submit (or delayed or any of the high-level APIs). So if you submit new work that's identical to something already running in the cluster, it will not repeat the task, but wait for the completion of the already-running one. (You can turn off this behaviour by using keyword pure=False )

Here we get the same random number four times:

In [35]: client.gather([client.submit(random.random) for _ in range(4)])
Out[35]: [0.609188678994237, 0.609188678994237, 0.609188678994237, 0.609188678994237]