[英]Airflow- Using TriggerDAGRunOperator to trigger DAG of different schedule
I'm new to Airflow.我是 Airflow 的新手。 I need to come up with a clean easy solution for DAG dependencies with different schedules.我需要为具有不同时间表的 DAG 依赖项提出一个干净简单的解决方案。
I have DAG 1 running Daily and DAG 2 - Weekly.我有 DAG 1 每天运行,DAG 2 - 每周运行。 How do I use TriggerDAGRunOperator to trigger weekly DAG from Daily one?如何使用 TriggerDAGRunOperator 从 Daily one 触发每周 DAG?
DAG 1:第 1 天:
with DAG('DAG 1',
schedule_interval='0 10 * * *'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TRG_TASK=TriggerDAGRunOperator(task_id='TRG_TASK',trigger_dag_id='DAG 2')
TASK1 >> TRG_TASK
DAG 2:第 2 天:
with DAG('DAG 2',
schedule_interval='15 10 * * 5'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
I know I can use ExternalTaskSensor Operator and mention timedelta, but it would become messy in long run.我知道我可以使用 ExternalTaskSensor Operator 并提及 timedelta,但从长远来看它会变得混乱。
Is there any easy/clean option with TriggerDAGRunOperator to check everyday if DAG 2 is indeed scheduled to run for that day then only trigger it else skip it on other days? TriggerDAGRunOperator 是否有任何简单/干净的选项来每天检查 DAG 2 是否确实计划在当天运行,然后只触发它,否则在其他日子跳过它?
Thanks谢谢
Is DAG2
always going to be triggered by DAG1
and you only want DAG2
to run weekly or are you expecting DAG2
will be executed weekly via the cron schedule you listed or triggered by DAG1
? DAG2
是否总是由DAG1
触发,您只希望DAG2
每周运行一次,还是希望DAG2
每周通过您列出或由DAG1
触发的 cron 计划执行?
If the expectation is DAG2
only gets triggered by DAG1
but you only want DAG2
to execute weekly, you can use the ShortCircuitOperator
in either DAG1
or DAG2
and update the schedule_interval
of DAG2
to None
(which means the DAG will only ever be triggered -- either manually or programmatically).如果期望DAG2
仅由DAG1
触发,但您只希望DAG2
每周执行一次,则可以在DAG1
或DAG2
中使用ShortCircuitOperator
并将DAG2
的schedule_interval
更新为None
(这意味着 DAG 只会被触发 - 要么手动或编程)。
Option 1: In DAG1
, add a ShortCircuitOperator
task prior to executing the TriggerDagRunOperator
which checks to see if the current day is the desired day of the week.选项 1:在DAG1
中,在执行TriggerDagRunOperator
之前添加一个ShortCircuitOperator
任务,该任务检查当前日期是否是所需的星期几。 If it is, then continue on with triggering DAG2
.如果是,则继续触发DAG2
。
Option 2: In DAG2
, add the ShortCircuitOperator
at the beginning of the workflow that performs the same day of week check.选项 2:在DAG2
中,在执行一周中同一天检查的工作流的开头添加ShortCircuitOperator
。 Again, if the desired day of week then continue on with the rest of the DAG.同样,如果需要一周中的某一天,则继续使用 DAG 的 rest。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.