Slurm- transfer files, local scratch

Question

How can I transfer the created files on the local scratch of each node (without logging into them) to somewhere else in Slurm? I am looking for something like a reversed sbcast .

Answer 1

Slurm's reversed sbcast is sgather . An example on how to use it with 10 tasks launched on 5 different nodes

$ salloc -N 5 --exclusive
$ srun -n 10 -N 5 bash -c 'echo $(hostname):$SLURM_JOB_ID:$SLURM_TASK_PID >> /tmp/file'
$ sgather /tmp/file ./file
$ ls
file.node1273 file.node1274 file.node1275 file.node1276 file.node1285
$ cat file.node1273
node1273:246987:1777529
node1273:246987:1777528

See man 1 sgather for details about options.