如何在特定目录上手动运行Airflow DAG

Question

I am evaluating whether Airflow is suitable for my needs (in bioinformatics). I am having some difficulty with the Airflow model. Specifically:

• Where does the DAG file actually get executed? What is its context? How can I pass input data into the DAG definition file? (Eg, I want ot create a task for each file in a directory.)
• How do I execute a DAG on an ad hoc basis? How do I pass parameters for the DAG construction?

Here is an example of what I would like to execute. Say I just received some data as a directory containing 20 files available in some shared filesystem. I want to execute a DAG pipeline which runs a particular bash command on each of the 20 files, then combines some of the results and performs further processing. The DAG needs the path on the filesystem and also to list the files in the directory to construct a task for each one.

It's probably not necessary for me to pass metadata from one task to another (which I understand is possible through XCom ), as long as I can dynamically construct the entire DAG upfront. But it's not clear to me how I can pass a path to the DAG construction.

Put another way, I'd like my DAG definition to include something like

dag = DAG(...)
for file in glob(input_path):
    t = BashOperator(..., dag=dag)

How do I get input_path passed in when I want to manually trigger a DAG?

I also don't really have need for the cron-style scheduling.

Answer 1

Regarding input_path you can pass it to the DAG using Airflow variables. Example of code used in the DAG file:

input_path = Variable.get("INPUT_PATH")

Variables can be imported using Airflow cli or manually through the UI.

You should use a subdag for this type of logic:

dag = DAG(...) for file in glob(input_path): t = BashOperator(..., dag=dag)

SubDAGs are perfect for repeating patterns. Defining a function that returns a DAG object is a nice design pattern when using Airflow.