[英]Airflow - xcom value acess into custom operator
I am using Airlfow, since last 6 months. 最近六个月以来,我一直在使用Airlfow。 I felt so happy to define the workflows in Airflow.
我非常高兴在Airflow中定义工作流程。 I have the below scenario where I am not able to get the xcom value (highlighted in yellow color).
我在以下情况下无法获得xcom值(以黄色突出显示)。
Please find the code below sample code: 请在下面的示例代码中找到代码:
def push_function(**context):
context['ti'].xcom_push(key='reportid', value='xyz')
dummy_operator = DummyOperator(
task_id='Start',
dag=main_dag
)
push_function_task = PythonOperator(
task_id='push_function',
provide_context=True,
python_callable=push_function,
op_kwargs={},
dag=main_dag)
push_function_task .set_upstream(dummy_operator)
custom_task = CustomOperator(
dag=main_dag,
task_id='import_data',
provide_context=True,
url="http://www.google.com/{}".format("{{task_instance.xcom_pull(task_ids='push_function')}}")
)
custom_task .set_upstream(push_function_task)
Notes : 1. CustomOperator is my own operator wtritten for downloading the data for the given URL 注意 :1. CustomOperator是我自己的操作员,负责下载给定URL的数据
Please help me. 请帮我。
Thanks, Samanth 谢谢,萨曼斯
I believe you have a mismatch in keys when pushing and pulling the XCom. 我相信您在推拉XCom时按键不匹配。 Each XCom value is tied to a DAG ID, task ID, and key.
每个XCom值都与DAG ID,任务ID和密钥相关联。 If you are pushing with
report_id
key, then you need to pull with it as well. 如果要使用
report_id
键推送,则也需要使用它。
Note, if a key is not specified to xcom_pull()
, it uses the default of return_value
. 注意,如果未为
xcom_pull()
指定键,则它将使用默认值return_value
。 This is because if a task returns a result, Airflow will automatically push it to XCom under the return_value
key. 这是因为如果任务返回结果,Airflow会在
return_value
键下将其自动推送到XCom。
This gives you two options to fix your issue: 这为您提供了两种解决问题的方法:
1) Continue to push to the report_id
key and make sure you pull from it as well 1)继续按到
report_id
键,并确保也从中拉出
def push_function(**context):
context['ti'].xcom_push(key='reportid', value='xyz')
...
custom_task = CustomOperator(
...
url="http://www.google.com/{}".format("{{ task_instance.xcom_pull(task_ids='push_function', key='reportid') }}")
)
2) Have push_function()
return the value you want to push to XCom, then pull from the default key. 2)让
push_function()
返回要推送到XCom的值,然后从默认键中拉出。
def push_function(**context):
return 'xyz'
...
custom_task = CustomOperator(
...
url="http://www.google.com/{}".format("{{ task_instance.xcom_pull(task_ids='push_function') }}")
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.