Airflow DAG Multiple Runs

Question

I have a DAG that I want to run multiple times after each successful completion. For an example I want to run it 10 times and stop. Is there a way to accomplish this? I tried looking into scheduling with CRON but it doesn't seem clean nor triggering the DAG via UI multiple times doesn't work (runs in parallel).

Answer 1

• You can, in addition to supplying a start_date , provide your DAG an end_date
• Quoting the docstring

:param start_date : The timestamp from which the scheduler will attempt to backfill

:type start_date : datetime.datetime

:param end_date: A date beyond which your DAG won't run, leave to None for open ended scheduling

:type end_date : datetime.datetime

---

While unrelated, also have a look at following scheduler settings in airflow.cfg as mentioned in this article

• run_duration
• num_runs

---

UPDATE-1

• In his article Use apache airflow to run task exactly once , @Andreas P has described a clever technique, which I believe can be adapted to your use-case. While even that won't be a very-tidy solution, it would at-least allow you to specify beforehand the number of runs (integer) for DAG instead of end_date .

• Alternatively (assuming you implement the above approach) rather than rather than baking this skipping-dag-after max-runs functionality within each DAG, you can create a separate orchestrator DAG that disables a given DAG after its max runs have passed.

Answer 2

You have to set property depends_on_past . This is set under DAG's default arguments section and it refers to previous instance dag instance. This is fix your problem.

Answer 3

I found a solution to my use case. It incorporated using depends_on_past=True ( mentioned by @Hitesh Gupta ) and setting your airflow.cfg file below:

# The maximum number of active DAG runs per DAG max_active_runs_per_dag = 1

This allowed us to only have one active DAG run at a time and also to not continue the next DAG run if there were failure in the previous run. This is for Airflow version 1.10.1 that I tested on.