如何在 Python 脚本中获取 PBS/Torque/Slurm 作业完成状态

Question

I am writing a python script which prepares input files for hundreds of jobs (say job.id = 1 to job.id = 1000, where job.id is a self-assigned id) and then submit them on a cluster for execution. Each job has 3 stages, s1, s2 and s3, where s2 is dependent on the results of s1 and s3 is dependent on results of s2. Each job may take 3 to 4 days using 48-64 cpu cores on super-cluster. I want my script to automatically handle all the stages for each job. One way that I thought is to submit s1 stage for all jobs at once and then periodically check the status of either output files (if exists) for all jobs or read in the queue status and see if a particular job disappeared from the queue (ie gets completed), after each 5 or 10 or 12 hours. A basic layout of my script is as follows.

import sched, time
from subprocess import *

jobs_running = True
s = sched.scheduler(time.time, time.sleep)

def Prepare():
    print "prepare jobs by reading some source files"
    print "set some flages for each job, e.g. job.id, job.stage, etc."
    print "submit jobs using < Popen('qsub nNodes Ncores jobinputfile') > "

def JobStatus():
    global jobs_running
    print "check status of each job"
    """
    for job in jobs:
        if job.stage1 == complete:
           print "goto stage 2"
           print "reset job.stage flages etc."
         elif job.stage2 == complete:
           print " go to stage 3"
           .
           .
         else last stage:

    if all stages complete for all jobs:
       set (global var) jobs_running = False
    """

def SecondStage():
    print " prepare for second stage "
    print " submit using < Popen('qsub nNodes Ncores jobinputfile') > "

def TimeSchedular(sc): 
    global jobs_running
    JobStatus()
    if jobs_running :
        s.enter(36000, 1, TimeSchedular, (sc,))

if (__name__ == "__main__"):    
    Prepare()
    s.enter(36000, 1, TimeSchedular, (s,))
    s.run()

This definitely is not an elegant solution for many reasons. For example, I have to check the status of every job in each cycle. Also if a job gets completed right after checking the status, it will be waiting for next 5 or 10 or 12 hours, to be submitted for the next stage. So my question is:

Is there some way to directly get job completion signal from PBS/SLURM or from the system in the above layout for, say job.id = 99, so that it can go to next stage (with out checking the status of the rest of the jobs) ? Or can someone suggest a better solution?

Answer 1

The normal way to accomplish this is through job dependencies . For example, if you have a job that depends on a another job before it can start, you can do something like this:

jobid1=`qsub phase_one.sh`
jobid2=`qsub phase_two.sh -W depend=afterok:${jobid1}`
# and so on as needed

The link there goes to the Torque documentation. I'm fairly certain that most any resource manager offers similar functionality.