How to limit usage of disk space in Snakemake?

Question

I work with 8 paired-end fastq files with 150 GB each, which need to be processed by a pipeline with space-demanding sub-tasks. I tried several options but I am still running out out disk space:

• used temp to delete output files when not needed anymore
• used disk_mb resources to limit number of parallel jobs.

I use the following execution to limit my disk space usage to 500GB, but apparently this is not guaranteed and exceeds the 500GB. How to limit the disk usage to a fixed value to avoid running out of disk space ?

snakemake --resources disk_mb=500000 --use-conda --cores 16  -p

rule merge:
  input:
    fw="{sample}_1.fq.gz",
    rv="{sample}_2.fq.gz",
  output:
    temp("{sample}.assembled.fastq")
  resources:
    disk_mb=100000
  threads: 16
  shell:
    """
    merger-tool -f {input.fw} -r {input.rv} -o {output}
    """


rule filter:
  input:
    "{sample}.assembled.fastq"
  output:
    temp("{sample}.assembled.filtered.fastq")
  resources:
    disk_mb=100000
  shell:
    """
    filter-tool {input} {output}
    """


rule mapping:
  input:
    "{sample}.assembled.filtered.fastq"
  output:
    "{sample}_mapping_table.txt"
  resources:
    disk_mb=100000
  shell:
    """
    mapping-tool {input} {output}
    """

Answer 1

Snakemake does not have the functionality to constrain resources, but can only schedule jobs in a way that respects resource constraints.

Now, snakemake uses resources to limit concurrent jobs, while your problem has a cumulative aspect to it. Taking a look at this answer , one way to resolve this is to introduce priority , so that downstream tasks have highest priority.

In your particular file, it seems that adding priority to the mapping rule should be sufficient:

rule mapping:
    input:
        "{sample}.assembled.filtered.fastq"
    output:
        "{sample}_mapping_table.txt"
    resources:
        disk_mb=100_000
    priority: 100
    shell:
        """
        mapping-tool {input} {output}
        """

You might also want to be careful about launching the rule initially (to avoid filling up the disk space with results of merge ).