在Luigi中重用通用任务

Question

I'm having trouble understanding how to make re-usable tasks in Luigi, and then use them in a concrete situation.

For example. I have two generic tasks that do something to a file and then output the result:

class GffFilter(luigi.Task):
    "Filters a GFF file to only one feature"
    feature = luigi.Parameter()
    out_file = luigi.Parameter()
    in_file = luigi.Parameter()
    ...

class BgZip(luigi.Task):
    "bgZips a file"
    out_file = luigi.Parameter()
    in_file = luigi.Parameter()
    ...

Now, I want a workflow that first filters, then bgzips a specific file using these tasks:

class FilterSomeFile(luigi.WrapperTask):
    def requires(self):
        return GffFilter(in_file='some.gff3', out_file='some.genes.gff3', filter='gene')

    def output(self):
        return self.inputs()

class BgZipSomeFile(luigi.Task):
    def run(self):
        filtered = FilterSomeFile()
        BzZip(filtered)

But this is awkward. In the first task I have no run method, and I'm just using dependencies to use the generic task. Is this correct? Should I be using inheritance here instead?

Then in the second task, I can't use dependencies, because I need the output from FilterSomeFile in order to use BgZip . But using dynamic dependencies seems wrong, because luigi can't build a proper dependency graph.

How should I make a Luigi workflow out of my generic tasks?

Answer 1

But this is awkward. In the first task I have no run method, and I'm just using dependencies to use the generic task. Is this correct?

Yes, according to this page , the WrapperTask is a dummy task whose purpose is to define a workflow of tasks, therefore it doesn't perform any actions by itself. Instead, by defining several requirements, this task will be complete when every requirement, listed in the requires method, has been completed. The main difference of this WrapperTask to a regular Task , is that you don't need to define an output method to signal that this task suceeded, as can be seen here .

Then in the second task, I can't use dependencies, because I need the output from FilterSomeFile in order to use BgZip. But using dynamic dependencies seems wrong, because luigi can't build a proper dependency graph.

Theoretically, you could make FilterSomeFile have the same output as GffFilter , make the BgZipSomeFile require FilterSomeFile , and then use the FilterSomeFile.output() in BgZipSomeFile.run to access the the zipped file. However, this solution would be somewhat strange because:

• The wrapper task only "runs" 1 other task, so the wrapped task could be used directly, without having to create a WrapperTask . A better usage of WrapperTask would involve merging BgZipSomeFile and FilterSomeFile in a single subclass of WrapperTask

• A Task is being instantiated in a run method. This results in a dynamic dependency, but this is not needed in this problem.

• Finally, the input of GffFilter is hardcoded in FilterSomeFile task, which makes the workflow less useful. This could be avoided by making the WrapperClass still receive parameters, and pass these parameters to its requirements.

A better solution would be:

import luigi as lg

class A(lg.Task):
    inFile = lg.Parameter()
    outFile = lg.Parameter()

    def run(self,):
        with open(self.inFile, "r") as oldFile:
            text = oldFile.read()

        text  += "*" * 10 + "\n" + "This text was added by task A.\n" + "*" * 10 + "\n"
        print(text)
        with open(self.outFile, "w") as newFile:
            newFile.write(text)

    def output(self,):
        return lg.LocalTarget(self.outFile)

class B(lg.Task):
    inFile = lg.Parameter()
    outFile = lg.Parameter()

    def run(self,):
        with open(self.inFile, "r") as oldFile:
            text = oldFile.read()

        text  += "*" * 10 + "\n" + "This text was added by task B.\n" + "*" * 10 + "\n"

        with open(self.outFile, "w") as newFile:
            newFile.write(text)

    def output(self,):
        return lg.LocalTarget(self.outFile)

class CustomWorkflow(lg.WrapperTask):
    mainOutFile = lg.Parameter()
    mainInFile = lg.Parameter()
    tempFile = "/tmp/myTempFile.txt"
    def requires(self,):
        return [    A(inFile = self.mainInFile, outFile = self.tempFile),
                    B(inFile = self.tempFile, outFile = self.mainOutFile)
                ]

This code can be run in command line with:

PYTHONPATH='.' luigi --module pythonModuleContainingTheTasks --local-scheduler CustomWorkflow --mainInFile ./text.txt --mainOutFile ./procText.txt