如何在 Airflow 中的不同 DAG 之间设置优先级

Question

Let's say we have two DAGs, dag1 and dag2, they serve different business requirements. they are completely unrelated. but dag1 is more important to have it finished as early as possible.
For simplicty, they both have only one task and they run daily.

In a scenario, where dag1 is behind the schedule with 2 or 3 days, I want to make sure that dag1 runs and completes its dag_runs first, ie dag1 is up to date following that dag2 is able to proceed.

I tried priority_weight but it doesn't work across different dags.

I need a way of putting those tasks from both different dags at the same queue and achieving DAG-level prioritization.

Answer 1

From the official documentation for the External Task Sensor :

Waits for a different DAG or a task in a different DAG to complete for
a specific execution_date.

    :param external_dag_id: The dag_id that contains the task you want to
        wait for
    :type external_dag_id: str
    :param external_task_id: The task_id that contains the task you want to
        wait for. If ``None`` the sensor waits for the DAG
    :type external_task_id: str
    :param allowed_states: list of allowed states, default is ``['success']``
    :type allowed_states: list
    :param execution_delta: time difference with the previous execution to
        look at, the default is the same execution_date as the current task or DAG.
        For yesterday, use [positive!] datetime.timedelta(days=1). Either
        execution_delta or execution_date_fn can be passed to
        ExternalTaskSensor, but not both.
    :type execution_delta: datetime.timedelta
    :param execution_date_fn: function that receives the current execution date
        and returns the desired execution dates to query. Either execution_delta
        or execution_date_fn can be passed to ExternalTaskSensor, but not both.
    :type execution_date_fn: callable
    :param check_existence: Set to `True` to check if the external task exists (when
        external_task_id is not None) or check if the DAG to wait for exists (when
        external_task_id is None), and immediately cease waiting if the external task
        or DAG does not exist (default value: False).
    :type check_existence: bool

Both DAGs should have depends_on_past Trigger Rule set to True so that the newer scheduled DAG runs will only execute if the previous scheduled runs have completed successfully.

Then add the External Task Sensor at the beginning of Dag 2 ( the one which executes later ).

Alternatively you could create your own custom sensor and use it via Airflow Plugins in order to check the metadatabase for the status of Dag Runs.

You could also build customer sensors that utilise either Airflow XCOMs or Airflow Variables to pass execution run times or any other Airflow Macro to a Sensor in DAG 2.

Answer 2

I find an ad-hoc solution where I could simply wrap both dags within a locking layer.

By that I mean, we add a simple task at the start that locks a specific row in a database and then at the end of the dag, we also add a simple task that unlocks the locked row.
Therefore when one of both dags is currently executing and the other wants to get started, it simply gets blocked because it cannot lock the specific row that is known for both dags.

Below is a simple description of the locking layer
dag1: lock_operator, task1, unlock_operator.
dag2: lock_operator, task1, unlock_operator.

Of course, we could let the lock_operator fail if it can not lock the row, and set retries_count to be very high so that we guarantee, it will still retry to lock until it can.

Answer 3

Your question is a bit vague as what you need is a dependency management method that for example an ExternalTaskSensor will do the trick, but what you want is a queue management with priority to make a dag (or a group of dags) special and get the hardware resources when they enter a queue in FIFO manner for example.

For priority you need to first use one of the queueing architectures of Airflow (AKA parallel processing plus using r3rd party apps for queue management) like RabbitMQ+Celery or Redis+Celery and then create different queus and assign your group A dags to the queue1 and group B dags to queue2 and later in settings change resource planings for each queue.

https://www.rabbitmq.com/priority.html

Good luck