使用Python子进程运行SLURM脚本以将多个长作业提交到队列中，并在继续python脚本之前等待作业完成

Question

Basics of the situation are that I'm using Python's subprocess module to run a SLURM script which submits a number of jobs to a queue on some HPC resources that I use. I would like the Python script to wait until all the jobs submitted are finished. I tried doing this with:

proc = subprocess.Popen(['sbatch slurm.sh'],shell=True)
proc.wait()

However, it only waits for the jobs to be submitted and not for all the jobs to finish. Anyone have any suggestions for how to make this work?

Note: One idea I had was potentially having subprocess communicate somehow with the .out file produced by the SLURM script (and wait for that to finish being edited maybe?), but I'm not sure how that would work.

Thanks for any ideas!

Answer 1

Use a python library to interface with Slurm, such as

• gc3pie
• joblib

and you will be able to manage the slurm jobs in a very easy way.

Answer 2

I have recently worked on slurm and have used -W or --wait parameter of sbatch to make the sbatch command first complete its execution and then execute the python code. Something like this sbatch -W slurm.sh