Dynamic task generation in an Airflow DAG

Question

I want to use Airflow to generate client reports, I would like to have one DAG that loops through all clients and launches a task to generate their report. The list of clients is gathered by the first task in the DAG and cannot be hardcoded in.

Basically I have a task that uses a database hook in a python operator to generate a list. Then for each item in the list I would like to execute a task using a python operator with that item being passed as an argument to the python function. Is there a certain pattern I can use to achieve this?

Answer 1

Trigger_dag concept

Let the task that uses a database hook in a python operator to generate a list" be the task in the controller dag and pass the each item in list to the trigger_dag in the params section.

You will find reference in the examples folder in your airflow installation

Good Luck!