批量提交snakemake作业

Question

I have a Snakefile that runs a python script which outputs many files in a directory. I wrote the following Snakefile script to execute this

MODELS = ["A", "B", "C"]
SEEDS = [1, 2, 3]

rule all: 
    input:
        expand("outputs/{model}/seed{seed}", model=MODELS, seed=SEEDS)

rule sub: 
    input: 
        {model}.py
    output: 
        directory("outputs/{model}/seed{seed}")
    run: 
        command = "python3 {} --seed {}".format(input, wildcards.seed)
        shell(command) 

Each python script files A.py , B.py , and C.py executes for hours. I want to be able to use sbatch that submits job without waiting for it to finish executing.

$ snakemake --cluster "sbatch --job-name=snakemake" --jobs 200 --latency-wait 1000

When I execute the following command, some files do not get run and Snakemake does not terminate. I tried writing a bash script that contains the above snakemake command and executed sbatch script.sh but that did not submit jobs in the Snakefile.

Is there a way to do this without snakemake waiting for sbatch jobs to finish executing?

Answer 1

It would not be a perfect solution, but have you tried the --immediate-submit flag? It will just submit all jobs to slurm with no regard for dependencies and without waiting. Normally, you'd supply a wrapper for sbatch that would translate snakemake dependencies to SLURM dependencies, but for a simple workflow like this (ie only one rule run many times), you might get away without that.