提交“SLURM”作业时出现“nohup”问题

Question

I have a python code main.py that runs bash script, the bash script inturn submits a job job.bash and obtains its JOBID using echo $JOBID | awk {'print $4'} echo $JOBID | awk {'print $4'} . If I run python in the terminal, the bash script works and I am able to obtain and echo the JOBID as follows:

#!/bin/bash 
JOBID=`sbatch ~/job.bash  | tee  output.log`
JOBID=`echo $JOBID | awk {'print $4'}`
echo $JOBID

Running above as part of python works in terminal python main.py , but doing nohup python main.py & , the echo does not print or store JOBID .

Any reason for this?

---

I am submitting a slurm job hence the JOBID is the pid from slurm

---

(Update Jul 17) Looks like the issue is with the command sbatch ~/job.bash | tee output.log sbatch ~/job.bash | tee output.log , it doesnt get submitted using nohup and hence JOBID never gets stored and echo'd.

(Update Jul 18) As per the comments from @pynexj adding set -x in the script results:

nohup: ignoring input and redirecting stderr to stdout
+ date
Mon Jul 18 21:46:35 +03 2022
++ sbatch ~/job.bash
++ tee output.log
+ JOBID=
++ echo
++ awk '{print $4}'
+ JOBID=
+ echo

The issue still persists. It appears that nohup is incompatible with sbatch .

---

Question: Why should nohup prevent submission of slurm job? Its objective is merely to capture terminate signal?

Answer 1

If this problem only happens with nohup present, you can get the benefits of nohup without actually using it with:

yourscript </dev/null >file.log 2>&1 & disown -h "$!"

This does the following:

• Redirects stdin from /dev/null with </dev/null
• Redirects stdout and stderr to a log file with >file.log 2>&1
• Tells the shell not to forward HUP signals to the background process with disown -h "$!"

...which is everything nohup does.