如何制作需要处理今天数据的DAG？

Question

I have a DAG that starts at 7:30pm every day. It needs to process the files located in /data/yyyy-mm-dd/ directory. yyyy-mm-dd is that same day.

If I use execution_date + timedelta(day=1) it works when the DAG is ran by the scheduler. But this breaks when I use the backfill command (I have to give it 2019-01-01 to run for 2019-01-02)

Is there a better way to accomplish this?

Answer 1

Your question sounds a little confused about the execution_date for backfills. The backfill command asks you to specify the alternate start and end dates to run the DAG in. It then uses the schedule_interval to figure out runs that would have run in that range and passes them their execution_date .

So, your schedule_interval probably looks like 30 19 * * * . And as you know your run is passed the start of the interval at the closing of that interval, so a scheduled execution_date of 2019-01-01T19:30:00.000 will be triggered to start after 2019-01-02T19:30:00.000. It seems at that time you want the job to pick up data that landed in /data/2019-01-02/ which is why you're adding a day to the execution_date and formatting it for the source.

If you're backfill ing, it should behave the same way (rather than shifting time around). So given -s 2019-01-01 -e 2019-01-02 it's going to backfill a run that would have been triggered after 2019-01-02T19:30:00.000 with the execution date of 2019-01-01T19:30:00.000 isn't it?

As for other ways to do this:

• You could move your runs to midnight and have them use the date in the execution_date . But 4.5h delay is probably not what you had in mind.
• You see if the data directories could be named differently, I doubt that would be okay if there's other people or jobs relying on them.
• Airflow also has a next_execution_date , which is basically going to give you the same result as adding a day to the execution_date . But you might like the formatted macro {{ next_ds }} for your needs.