使用 SLURM 在集群上运行作业时如何保存 output

Question

I want to run an R script using SLURM. I have created the R script, "test.R" as shown:

print("Running the test script")
write.csv(head(mtcars), "mtcars_data_test.csv")

I created a bash script to run this R script "submit.sh"

#!/bin/bash

#sbatch --job-name=test.job
#sbatch --output=.out/abc.out
Rscript  /home/abc/job_sub_test/test.R

And I submitted the job on the cluster

sbatch submit.sh

I am not sure where my output is saved. I looked in the home directory but no output file.

Edit

I also set my working directory in test.R , but nothing different.

setwd("/home/abc")
print("Running the test script")
write.csv(head(mtcars), "mtcars_data_test.csv")

When I run the script without SLURM Rscript test.R , it worked fine and saved the output according to the set path.

Answer 1

Slurm will set the job working directory to the directory which was the working directory when the sbatch command was issued.

Assuming the /home directory is mounted on all compute nodes, you can change explicitly the working directory with cd in the submission script, or setwd() in the R syntax. But that should not be necessary.

Three possibilities:

• either the job did not start at all because of a configuration or hardware issue; that you can find out with the sacct command, looking at the state column.
• either the file was indeed created but on the compute node on a filesystem that is not shared; in that case the best option is to SSH to the compute node (which you can find out with sacct ) and look for the file there; or
• the script crashed and the file was not created at all, in that case you should look into the output file of the job ( .out/abc.out ). Beware that the .out directory must be present before the job starts, and that, as it starts with a . , it will be a hidden file, revealed in ls only with the -a argument.

Answer 2

The --output argument to sbatch is relative to the folder you submitted the job from. setwd inside the R script wouldn't affect it, because Slurm has already parsed that argument and started piping output to the file by the time the R script is running.

First, if you want the output to go to /home/abc/.out/ make sure you're in your homedir when you submit the script, or specify the full path to the --output argument.

Second, the.out folder has to exist; I tested this and Slurm does not create it if it doesn't.