使用 Airflow 中的另一个 dag 触发外部 dag

Question

Having list of tasks which calls different dags in master dag.I'm using the TriggerDagrunoperator to accomplish this. But facing few issues.

• TriggerDagrunoperator doesn't wait for completion of external dag, it triggers next task. I want that to wait until completion and next task should trigger based on the status. Came across ExternalTaskSensor. It is making the process complicated. Is there any other solution to fix this?

• If I trigger the master dag again, I want the task to restart from where it is failed. Right now, it's not restarting, but for time based schedule,it will.

Answer 1

.. I want that to wait until completion .. Came across ExternalTaskSensor. It is making the process complicated ..

I'm unaware of any other way to achieve this. I myself did this the same way .

---

If I trigger the master dag again, I want the task to restart from where it is failed...

This requirement of your goes against the principle of idempotency that Airflow demands. I'd suggest you try to re-work you jobs in incorporate idempotency (for instance in case of retries, you have to have idempotency). Meanwhile you can take inspiration from some people and try to achieve something similar (but it will be pretty complicated)

Answer 2

With Airflow 2.0.1, the triggering dag can be made to wait for completion of target dag with parameter wait_for_completion

ref: here