在分配给其他节点之前，无法让 slurm sbatch 作业数组将作业分配给核心

Question

I have a number of jobs that require a single core to run. The cluster I use has 5 nodes, each with 96 cores. When I use slurm to submit the jobs, the jobs are always assigned to multiple nodes and if more than 5 (ie, number of nodes) they tend to run sequentially as opposed to concurrently on each node. The same behaviour is observed when I restrict the nodes; sequential, not concurrent. The configuration is set to "cons_tres" and I have tried many different suggestions and combinations of the script below. I did manage to get the desired operation using the $SLURM_PROCID accessed through a wrapper script, but I need to access data throughout the run for each model and have found the $SLURM_ARRAY_TASK_ID very convenient for this. I have tried submitting with srun within the sbatch script, but nothing seems to work. The last iteration with the optional srun inclusion is shown below. I am pretty new (~1 week) to the development of scheduling scripts, so please forgive any incorrect/inaccurate descriptions. I really appreciate any solutions, but am also looking to more fully understand where I am going wrong. Thanks!

#!/bin/tcsh
## SLURM TEST

#SBATCH --job-name=seatest
#SBATCH --nodes=1-1
#SBATCH --ntasks=5
#SBATCH --ntasks-per-node=5
#SBATCH --array=1-5
#SBATCH --output=slurm-%A_%03a.out

hostname

set CASE_NUM=`printf %03d $SLURM_ARRAY_TASK_ID`

[srun] program-name seatest.$CASE_NUM.in

This jobs were sent to 1 core of each of the five nodes, not to 5 cores of 1 node.

Answer 1

Memory based scheduling was enabled on the cluster, which required the memory (--mem) for each job to be specified.