[英]How to branch multiple paths in Airflow DAG using branch operator?
This is what I want, but I don't know how to achieve this in airflow, as both of the tasks are being executed.这就是我想要的,但我不知道如何在 airflow 中实现这一点,因为这两个任务都在执行。
To summarize:总结一下:
option_1 -> complete
or option_2 -> Do_x, Do_y -> complete
option_1 -> complete
或option_2 -> Do_x, Do_y -> complete
How should I structure this?我应该如何构建这个? I have this as my current code:
我有这个作为我当前的代码:
(t1 >> t2 >> option_1 >> complete)
(t1 >> t2 >> option_2 >> do_x >> do_y >> complete)
t2 in this case is a branch operator.在这种情况下,t2 是一个分支运算符。
I've also tried the syntax for ... [option_1, option_2]...
but I need a completely separate path to execute, not just a single task to be switched.我也尝试了
... [option_1, option_2]...
的语法,但我需要一个完全独立的路径来执行,而不仅仅是一个要切换的任务。
The dependancies you have in your code are correct for branching.您在代码中拥有的依赖项对于分支是正确的。 Make sure
BranchPythonOperator
returns the task_id
of the task at the start of the branch based on whatever logic you need.确保
BranchPythonOperator
根据您需要的任何逻辑在分支开始时返回任务的task_id
。 More info on the BranchPythonOperator
here .有关
BranchPythonOperator
的更多信息,请点击此处。 One last important note is related to the "complete" task.最后一个重要注意事项与“完成”任务有关。 Since branches converge on the "complete" task, make sure the
trigger_rule
is set to "none_failed" (you can also use the TriggerRule
class constant as well) so the task doesn't get skipped.由于分支会聚在“完成”任务上,因此请确保将
trigger_rule
设置为“none_failed”(您也可以使用TriggerRule
class 常量),这样任务就不会被跳过。
Quick code test for your reference:快速代码测试供您参考:
from airflow.models import DAG
from airflow.operators.dummy import DummyOperator
from airflow.operators.python import BranchPythonOperator
from airflow.utils.trigger_rule import TriggerRule
from datetime import datetime
DEFAULT_ARGS = dict(
start_date=datetime(2021, 5, 5),
owner="airflow",
retries=0,
)
DAG_ARGS = dict(
dag_id="multi_branch",
schedule_interval=None,
default_args=DEFAULT_ARGS,
catchup=False,
)
def random_branch():
from random import randint
return "option_1" if randint(1, 2) == 1 else "option_2"
with DAG(**DAG_ARGS) as dag:
t1 = DummyOperator(task_id="t1")
t2 = BranchPythonOperator(task_id="t2", python_callable=random_branch)
option_1 = DummyOperator(task_id="option_1")
option_2 = DummyOperator(task_id="option_2")
do_x = DummyOperator(task_id="do_x")
do_y = DummyOperator(task_id="do_y")
complete = DummyOperator(task_id="complete", trigger_rule=TriggerRule.NONE_FAILED)
t1 >> t2 >> option_1 >> complete
t1 >> t2 >> option_2 >> do_x >> do_y >> complete
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.