Snakemake 重建/重新安排工作

Question

I am writing a pipeline in Snakemake for people that don't have much programming knowledge, so I want to want them to be able to run the entire pipeline by only requesting snakemake all -c in the command line.

I have 2 config files in my Snakefile:

configfile: "config.yaml"
configfile: "config_samples.yaml"

These config files will be merged together by Snakemake.

config.yaml is the standard config file. config_samples.yaml is a config file of which its contents change depending on the pipeline input. It looks like the following:

samples:
  CYP20130000B:
    R1: CYP20130000B_R1.fastq
    R2: CYP20130000B_R2.fastq
  SAT20020000A:
    R1: SAT20020000A_R1.fastq
    R2: SAT20020000A_R2.fastq
  ...

I am using a Python script in a Snakemake rule to generate the contents of config_samples.yaml (using the snakemake script directive). This works fine.

However, when I list all my wanted output in the all rule like so:

"config_samples.done", # flag file for rule that generates config_samples.yaml
expand(QC_raw_reads/{sample}_{direction}_fastqc.html", sample=config["samples"], direction=["R1", "R2"])

Then this won't work, because the expand() will only expand to samples that are in the current config_samples.yaml , so before the Python script actually generates the new config_samples.yaml with the new samples.

This can easily be avoided by running the rule that generates config_samples.yaml separately before running the enterire pipeline, but to refer back to the beginning, I want it to stay as easy as possible for non-programmers.

So, I was wondering if there is a way to let Snakemake rebuild/reschedule the jobs, so that they can be updated for the new samples.

Answer 1

I haven't fully digested the questions and the comments underneath, but I feel agreeing with @DmitryKuzminov in that the current setup is a bit contrieved.

In any case, I think you can make snakemake to regenerate config_samples.yaml by adding towards the top of the Snakefile, before the first rule, something like:

if os.path.exists('config_samples.yaml'):
    os.remove('config_samples.yaml')

you could even make that step controllable on the command line with:

if config['remake_config'] == 'yes':
    if os.path.exists('config_samples.yaml'):
        os.remove('config_samples.yaml')

and then execute with:

snakemake -C remake_config='yes' ...

Answer 2

Your entire problem seems to come about because of this sentence:

I am using a Python script in a Snakemake rule to generate the contents of config_samples.yaml (using the snakemake script directive). This works fine.

The solution could be as simple as realising that the Python script could run outside of a Snakemake rule just at the beginning of your Snakefile. A Snakefile is basically a Python script. So you can just import your script into that Snakefile and execute it there.

So your Snakefile could look like this:

import config_generator
config = config_generator.add_samples(config)

rule all:
   input: expand(QC_raw_reads/{sample}_{direction}_fastqc.html", sample=config["samples"], direction=["R1", "R2"])

rule A:
...

config_generator.py would look like this:

def add_samples(config):
    # here you can calculate whatever you calculate to get config_samples
    # e.g. config["samples"] = ...
    return config