Dask jobqueue - 有没有办法同时启动所有工人？

Question

Say if I have the following deployment on SLURM:

cluster = SLURMCluster(processes=1, cores=25, walltime=1-00:00:00)

cluster.scale(20)
client = Client(cluster)

So I will have 20 nodes each with 25 cores.

Is there a way to tell the slurm scheduler to start all nodes at the same time, instead of starting each one individually when they become available?

A specific example: when nodes are being started individually, those that started the earliest might wait for several, say 2, hours until all 20 nodes are ready. This not only is a waste of resources but also this makes my total job time to be less than 24 hour (eg 22 hours).

This is something one can do easily with dask_mpi , where a single batch job is allocated. I am wondering if it's possible to do this with dask_jobqueue specifically.

Answer 1

dask-jobqueue itself doesn't propose such a functionality.

It is designed to submit independent jobs. So to achieve this you would have to look at the possibilities of the job queuing system, Slurm in your case, and see if this is possible without dask-jobqueue. Then you should try to add the correct options to dask-jobqueue if you can, though job_extra_directives kwarg for example.

I'm not aware of such a functionality within Slurm, but there are so many knobs it is hard to tell. I know this is not possible with PBS.

A good option to achieve what you want is, as you said so, using dask-mpi .

A final thought, you could also start your computation with the first two nodes, not waiting for the other to be ready. This should be doable in most cases.