如何动态创建 airflow 任务

Question

We have 100+ Airflow DAGs which runs daily on their scheduled time. Upon failure the DAGs alerts via Email. Our business team wants a notification when all 100+ DAGs are successfully completed for that day so they can analyze the report data.

One way is to create a DAG which will monitor all these 100+ DAGs and upon successful, will trigger an email to business team.

The problem with this approach is we need to have 100+ ExternalTaskSensor operator which will monitor all these DAGs, also from maintenance point of view its not good as number of DAGs keep on increasing.

We know that creating dynamic task is possible as per - How to dynamically create tasks in airflow

But how to iterate 100+ values(dag id) stored in Airflow CLI Variables in DAGs?

Answer 1

The best solution is storing the dag ids in a file accessible by Airflow. But if it is complicated or you want a fully dynamic way to achieve that, you can add a new tag to_monitor for the dags you want to monitor.

Here is an example for 4 dags to monitor and 4 other dags:

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from datetime import datetime

for i in range(4):
    dag_id = f'monitor_dag_{i}'
    with DAG(
            dag_id=dag_id,
            start_date=datetime(2022, 8, 27),
            tags=["to_monitor"]
    ) as dag:
        task = EmptyOperator(
            task_id='empty_task',
        )
        globals()[dag_id] = dag


for i in range(4):
    dag_id = f'no_monitor_dag_{i}'
    with DAG(
            dag_id=dag_id,
            start_date=datetime(2022, 8, 27),
    ) as dag:
        task = EmptyOperator(
            task_id='empty_task',
        )
        globals()[dag_id] = dag

Then you can use PostgresHook to access Airflow Metadat (if you are using another database, you should replace this by its hook) and query the tags table to get the dags ids which have the tag to_monitor . Finally you can create your sensors automatically:

from airflow.models import DAG
from airflow import XComArg
from airflow.operators.empty import EmptyOperator
from airflow.providers.postgres.hooks.postgres import PostgresHook
from airflow.sensors.external_task import ExternalTaskSensor
from datetime import datetime


def get_dags_ids_to_monitor():
    postgres_hook = PostgresHook()
    dags_ids = postgres_hook.get_records(
        sql="SELECT dag_id FROM dag_tag"
    )
    # convert list of tuples of one string to a list of strings
    return [dag_id[0] for dag_id in dags_ids]


with DAG(
    dag_id="monitor_dag",
    start_date=datetime(2022, 8, 27)
) as dag:

    dags_to_monitor = get_dags_ids_to_monitor()

    sensor_tasks = [
        ExternalTaskSensor(
            task_id=f"{external_dag_id}_external_tasks_sensor",
            external_task_id="empty_task",
            external_dag_id=external_dag_id
        )
        for external_dag_id in dags_to_monitor
    ]
    notify_task = EmptyOperator(task_id="send_notification")
    sensor_tasks >> notify_task

Here is the graph:

For your daily dags, you need just to add at the end of these dags, an Empty task finish to use it in the sensors ( empty_task in my example), and create a new connection with Airflow Metadata creds.

With this solution, Airflow will query the database every dag_dir_list_interval and on every task run, but don't worry it's similar to getting a variable in the task instance with Variable.get() .