在snakemake 中，如何在分散-收集过程中使用通配符？

Question

I am trying to use snakemake's scatter-gather functionality to parallelize a slow step in my workflow. However, I cannot figure out how to apply it in situations where I am using wildcards. For example, I have defined the wildcard library in rule all , however, this does not seem to apply to the scatter function in ScatterIntervals :

import re

SCATTER_COUNT = 100
scattergather: 
    split=SCATTER_COUNT

rule all:
    input:
        expand("{library}_output.txt", library=["FC19271512", "FC19271513"])


rule ScatterIntervals:
    input:
        "{library}_baits.interval_list"
    output:
        temp(scatter.split("tmp/{library}_baits.scatter_{scatteritem}.interval_list"))
    params:
        output_prefix = (
            lambda wildcards, output: 
            re.sub("\.scatter_\d+\.interval_list", "", output[0])
        ),
        scatter_count = SCATTER_COUNT
    shell:
        """
        python ScatterIntervals.py \
            -i {input} \
            -o {params.output_prefix} \
            -s {params.scatter_count}
        """


rule ProcessIntervals:
    input:
        bam = "{library}.bam",
        baits = "tmp/{library}_baits.scatter_{scatteritem}.interval_list"
    output:
        temp("tmp/{library}_output.scatter_{scatteritem}.txt")
    shell:
        """
        python ProcessIntervals.py \
            -b {input.bam} \
            -l {input.baits} \
            -o {output}
        """


rule GatherIntervals:
    input:
        gather.split("tmp/{library}_output.scatter_{scatteritem}.txt")
    output:
        "{library}_output.txt"
    run:
        inputs = "-i ".join(input)
        command = f"python GatherOutputs.py {inputs} -o {output[0]}"
        shell(command)
    

WildcardError in line 16 of Snakefile: 
No values given for wildcard 'library'.

Answer 1

Evidently this works like expand , in that you can quote the wildcards that aren't scatteritem if you want DAG resolution to deal with them:

temp(scatter.split("tmp/{{library}}_baits.scatter_{scatteritem}.interval_list"))

The same logic applies for gather.split .

Answer 2

I modify my code as mentioned like this.

rule fastq_fasta:
    input:rules.trimmomatic.output.out_file
    output:"data/trimmed/{sample}.fasta"
    shell:"sed -n '1~4s/^@/>/p;2~4p' {input} > {output}"

rule split:
    input:
        "data/trimmed/{sample}.fasta"
    params:
        scatter_count=config["scatter_count"],
        scatter_item = lambda wildcards: wildcards.scatteritem
    output:
        temp(scatter.split("data/trimmed/{{sample}}_{scatteritem}.fasta"))
    script:
        "scripts/split_files.py"
        
rule process:
    input:"data/trimmed/{sample}_{scatteritem}.fasta"
    output:"data/processed/{sample}_{scatteritem}.csv"
    script:
        "scripts/split_files.py"

rule gather:
    input:
        gather.split("data/processed/{{sample}}_{scatteritem}.csv")
    output:
        "data/processed/{sample}.csv"
    shell:
        "cat {input} > {output}"

However, i got AmbiguousRuleException: Rules fastq_to_fasta(which is previous rule) and split are ambiguous for the file data/trimmed/Ornek_411-of-81-of-81-of-81-of-81-of-81-of-81-of-81-of-81-of-8.fasta

I tried lots of things but either rules are not calling or take AmbiguousRuleException. What am i missing, can someone help?