Airflow DAG 激活，但有延迟

Question

So I made an Apache Airflow system in a Docker and so far it works perfectly well, with one problem, that persists through all dags: they activate on the previous iteration, not the current one.

For example, if I make a DAG that activates every minute, when it is 15:08, it will activate the DAG for 15:07. And if I make a DAG that activates every year, when it is 2023, it will activate the DAG for 2022, but not the current year.

Is there any way to fix this? Or is it supposed to be that way, and I should just account for this?

Here is the code for some of my dags as an example:

from datetime import datetime
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
import logging    
import random
import pandas as pd
import sqlalchemy
from airflow.utils.log.logging_mixin import LoggingMixin
from dateutil.relativedelta import relativedelta
  

import requests

from datetime import datetime

def test_print(ds, foo, **kwargs):
    start_date = str(ds)
    end_date = str((datetime.strptime(ds, '%Y-%m-%d') + relativedelta(years=1)).date())

    
    print('HOLIDAYS:')
    print('--------------')
    print('START DATE:' + start_date)
    print('END DATE:' + end_date)
    print('--------------')
    
    now = ds
    data2send = {'the_date_n_hour': now}
    
    r = requests.post("http://[BACKEND SERVER]:8199/do_work/",json=data2send)
    print(r.text)
    assert now in r.text
    
    task_logger = logging.getLogger('airflow.task')
    task_logger.warning(r.text)
    
    return 'ok'

dag = DAG('test_test', description='test DAG',
          schedule_interval='*/1 * * * *',
          start_date=datetime(2017, 3, 20), catchup=False)

test_operator = PythonOperator(task_id='test_task', 
                               python_callable=test_print, 
                               dag=dag,
                               provide_context = True,
                               op_kwargs={'foo': 'bar'})

test_operator

from __future__ import print_function

import time
from builtins import range
from pprint import pprint

import airflow
from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator

import sqlalchemy
import pandas as pd
import datetime
import requests

from dateutil.relativedelta import relativedelta

args = {
    'owner': 'airflow',
    "depends_on_past": False,
    "retries": 12,
    "retry_delay": datetime.timedelta(minutes=60)}
   
dag = DAG(
    dag_id='dag_holidays',
    default_args=args,
    schedule_interval='0 12 1 1 *',
    start_date=datetime.datetime(2013, 1, 1), 
    catchup=True)

def get_holidays(ds, gtp_id, **kwargs):
    """Wait a bit so that SQL isn't overwhelmed"""
    holi_start_date = str(ds)
    holi_end_date = str((datetime.strptime(ds, '%Y-%m-%d') + relativedelta(years=1)).date())

    
    print('HOLIDAYS:')
    print('--------------')
    print('GTP ID: {}'.format(str(gtp_id)))
    print('START DATE:' + holi_start_date)
    print('END DATE:' + holi_end_date)
    print('--------------')
    r = requests.post("http://[BACKEND SERVER]/load_holidays/",data={'gtp_id': gtp_id, 'start_date': holi_start_date, 'end_date': holi_end_date})
    if 'Error' in r.text:
        raise Exception(r.text)
    else:
        return r.text
    return ds

engine = sqlalchemy.create_engine('[SQL SERVER]')
query_string1 = f""" select gtp_id from gtps"""
all_ids = list(pd.read_sql_query(query_string1,engine).gtp_id)


for i, gtp_id in enumerate(all_ids):
    task = PythonOperator(
        task_id='holidays_' + str(gtp_id),
        python_callable=get_holidays,
        provide_context = True,
        op_kwargs={'gtp_id': gtp_id},
        dag=dag,
    )

    task

Answer 1

Yes, this is supposed to be this way and it can definitely be a bit confusing at first.

The reason for this behavior is that Airflow was used for a lot of ETL type processing when it was built and with that pattern you are running your DAG on the data of the previous interval.

For example when your data processing DAG runs every day at 3am, the data it processes is the data what was collected since 3am the previous day. This period is called the Data Interval in Airflow terms. The start of the data interval is the Logical Date (in earlier versions called execution date), which is what is incorporated into the Run ID. I think this is what you are seeing as the previous iteration. The end of the data interval is the Run After date, this is when the DAG actually will be scheduled to run.

When you hover over the Next Run: field in the Airflow UI for a given DAG you will see all of those dates and timestamps for the next run of a specific DAG.

This guide on scheduling DAGs might be helpful as a reference and it has some examples.

Disclaimer: I work for Astronomer, the company behind the guide I linked. :)