[英]How to use xCom in airflow dag file using python operator?
I'm working on this airflow dag file to do some test with XCOM, but not sure how to use it between python operators.我正在处理这个气流 dag 文件以对 XCOM 进行一些测试,但不确定如何在 python 运算符之间使用它。 Can someone please help how to write the logic to pass a message between the python operators using XCOM push and pull functions.
有人可以帮助如何编写逻辑以使用 XCOM 推拉功能在 python 运算符之间传递消息。 Below is the dag file that i'm working on...
下面是我正在处理的 dag 文件...
The question is how to pass a message from each task to another task问题是如何将消息从每个任务传递到另一个任务
@dag_factory
def create_dag():
with DAG(
dag_id="DSStest",
default_args=default_args,
schedule_interval=timedelta(1),
) as dag:
# Define operators here, for example:
output_file = path_in_workspace("testout")
rscript_file = path_in_workspace("rtest2.R ")
bcmd1 = "downloading some file here..."
t0 = PythonOperator(
task_id="start",
python_callable=my_func2,
provide_context=True,
op_args=[output_file, 0],
)
t1 = PythonOperator(
task_id="job1",
python_callable=my_func1,
provide_context=True,
op_args=[output_file, 1],
)
t2 = PythonOperator(
task_id="job2",
python_callable=my_func1,
provide_context=True,
op_args=[output_file, 2],
)
t10 = PythonOperator(
task_id="job10",
python_callable=my_func2,
provide_context=True,
op_args=[output_file, 10],
)
t20 = BashOperator(
task_id="job20",
bash_command=bcmd1,
queue={
"worker_type": "png-bash-worker",
"request_memory": "1G",
"request_cpu": 1,
},
)
# Define dependencies between operators here, for example:
t0 >> t1
t0 >> t2
t1 >> t10
t2 >> t10
t10 >> t11
t11 >> t20
return dag # Do not change this
I recommend to take a look at this example, it shows/explains everything regarding xcoms and PythonOperators:我建议看一下这个例子,它显示/解释了关于 xcoms 和 PythonOperators 的一切:
example_xcom.py example_xcom.py
The official airflow page also explains xcoms in a lot of detail:官方的airflow页面也对xcoms做了很多详细的解释:
official documentation官方文件
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.