切换到 ON 时，气流 dag 会自动触发 DAG？

Question

I have created a DAG with schedule interval "*/10 * * * *" . But when I toggle DAG to ON through Airflow UI (at the time 01/07/2020 07:50:00 ). Its automatically getting triggered without waiting to complete the specified interval.

#Specified start date
start_date : datetime.strptime('01/07/2020 06:35:00', '%m/%d/%Y %H:%M:%S')

I have tried by adding 'catchup': False to dag_args still facing the same problem

Answer 1

It's due to incorrect START_DATE you provided. Per the cron expression "*/10 * * * *", your dag will trigger every 10 minutes. When you unpaused the dag at the time 01/07/2020 07:50:00, run id 01/07/2020 07:40:00 got triggered immediately.

First, I recommend you use constants for start_date, because dynamic ones would act unpredictably based on with your airflow pipeline is evaluated by the scheduler.

More information about start_date here in an FAQ entry that I wrote and sort all this out: https://airflow.apache.org/faq.html#what-s-the-deal-with-start-date

Now, about execution_date and when it is triggered, this is a common gotcha for people onboarding on Airflow. Airflow sets execution_date based on the left bound of the schedule period it is covering, not based on when it fires (which would be the right bound of the period). When running an schedule='@hourly' task for instance, a task will fire every hour. The task that fires at 2pm will have an execution_date of 1pm because it assumes that you are processing the 1pm to 2pm time window at 2pm. Similarly, if you run a daily job, the run an with execution_date of 2016-01-01 would trigger soon after midnight on 2016-01-02.

This left-bound labelling makes a lot of sense when thinking in terms of ETL and differential loads, but gets confusing when thinking in terms of a simple, cron-like scheduler.