防止 Slurm 用户在他/她自己的工作上运行 scancel

Question

In our HPC setup where Slurm is installed, after a user logs in using ssh, a script is executed where a job is submitted using srun with an interactive bash. The script is placed on.profile and the user cannot modify it. This is the contents of the bash script file.

srun -A <ACCOUNT_NAME> -p --mem=< MEMORY> --time=HH:MM:SS -N <NUM_NODES> --gres=gpu:<NUM_GPU> --pty /bin/bash -i

Is there a way to prevent this user from bypassing slurm? Since the user can perform scancel on the job id anyway. Or is there a way to prevent the user from performing the scancel command?

Please advise. Thank you.

Answer 1

There is no simple way to prevent users from managing their own job (you could create wrappers that submit/manage jobs with another user id, but that would be difficult to manage/maintain/).

One option could be to add exit or logout after the srun... line in .profile so that as soon as the Slurm job is terminated (either scancel led or after the user has exit ed from the interactive session, the SSH session is terminated as well.

srun -A ...
exit

Another option is to exec the srun command rather than simply running it:

exec srun -A ...

so that the srun instance replaces the Bash instance running outside of the control of Slurm. As soon as srun is terminated (for either reason as above), the user will be disconnected.