LSF - 使用 sasbatch 脚本自动重新运行作业

Question

I am trying to create an auto-rerun mechanism by implementing some code into sasbatch script after sascommand will finish. General idea is to:

1. locate a log of sas process and an id of the flow containing current job,

2. check if the log contains particular ORA-xxxxx errors which we know that solution for them is just rerun of the process,

3. if so, then trigger jrerun class from LSF Platform Command Line Interface,

4. exit sasbatch passing $rc to LSF

The idea was implemented as:

#define used paths
log_dir=/path/to/sas_logs_directory
out_log=/path/to/auto-rerun_log.txt
out_log2=/path/to/lsf_rerun_log.txt

if [ -n "${LSB_JOBNAME}"]; then
    if [ ! -f "$out_log"]; then
        touch $out_log
    fi
    #get flow runtime attributes
    IFS-: read -r flow_id username flow_name job_name <<< "${LSB_JOBNAME}"

    #find log of the current process
    log_path=$(ls -t $log_dir/*.log | xargs grep -li "job:\s*$job_name" | grep -i "/$flow_name_" | head -1)

    #set path to txt file containing lines which represents ORA errors we look for
    conf_path-/path/to/error_list

    #analyse process' log line by line
    while read -r line;
    do
        #if error is found in log then try to rerun flow
        if grep -q "$line" $log_path; then
            (nohup /path/to/rerun_script.sh $flow_id >$out_log2 2>&1) &
            disown
            break
        fi
    done < $conf_path
fi

While rerun_script is the script which calls jrerun class after sleep command - in order to let parent script exit $rc in the meanwhile. It looks like:

sleep 10
/some/lsf/path/jrerun

Problem is that job is running for the all time. In LSF history I can see that jrerun was called before job exited. Furthermore in $out_log2 I can see message: <flow_id> has no starting or exit points.

Do anyone have an idea how I can pass return code to LSF before jrerun calling? Or maybe some simplier way to perform autorerun of SAS jobs in Platform LSF?

I am using SAS 9.4 and Platform Process Manager 9.1

Answer 1

Or maybe some simplier way to perform autorerun of SAS jobs in Platform LSF?

I'm not knowledgeable about the SAS part. But on the LSF side there's at least a couple of ways to requeue the job.

If you have control of the job script, you can use special process exit value to automatically requeue the job.

https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_requeue_about.html

If you have control outside of the job script, you can use brequeue -r to requeue a running job.

https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/brequeue.1.html

Good Luck

Answer 2

I managed to get this working by using two additional configuration files. When my grep returnes 1 I add found flow_id to flow_list.txt configuration file and modify especially made trigger_file.txt .

I scheduled additional flow execute_rerun in LSF which is triggered after file trigger_file.txt is modified. The execute_rerun flow reads flow_list.txt configuration file line by line and calls jrerun method on each flow.

I managed to achieve an automatic rerun of the flows which fails due to particular errors.