简体   繁体   English

如何在 Airflow 运算符中应用 Python 函数超过 Airflow XCom_pull 值

[英]How to apply Python Functions over Airflow XCom_pull value within Airflow Operator

I have a requirement where i need to capture XCOM response over Airflow SQLsensor operator and apply some python command to change data format.我有一个要求,我需要通过 Airflow SQLsensor 运算符捕获 XCOM 响应并应用一些 python 命令来更改数据格式。

code which i have is given below:-我的代码如下: -

delta_sql_sensor_task = SqlSensor( 
    task_id= "Verify-Completion-of-Delta-Job", 
    poke_interval= 10, 
    timeout=100,
    conn_id="lct_statedb",
    #gobal_id_list="{{ str(ti.xcom_pull(key='id_list', task_ids=['hook_task'])).replace('[','') }}",
    sql="SELECT COALESCE ( (SELECT count(*)  FROM lct_transformation_history WHERE deltaload_transformation_status is not null AND fullload_transformation_status is not null AND category='sites' AND id in (" + "{{ (ti.xcom_pull(key='id_list', task_ids=['hook_task'])) }}" + ") group by category),0) AS TRIGGER;",
    trigger_rule='one_failed',
    dag=dag)

Response which i receive from the code "{{ (ti.xcom_pull(key='id_list', task_ids=['hook_task'])).str() }}" is as ([('13',)],) but i need to change response response to '13'我从代码"{{ (ti.xcom_pull(key='id_list', task_ids=['hook_task'])).str() }}"收到的响应为([('13',)],)但我需要将响应响应更改为'13'

Following Options which i tried are given below:-我尝试过的以下选项如下: -

  1. I tried to store xcom_pull response in a Global variable gobal_id_list and apply Python functions but it didn't worked.我尝试将 xcom_pull 响应存储在全局变量gobal_id_list并应用 Python 函数,但没有成功。
  2. I also tried to apply Python operator within SQL variable as given below but no luck, code given below:- sql="SELECT COALESCE ( (SELECT count(*) FROM lct_transformation_history WHERE deltaload_transformation_status is not null AND fullload_transformation_status is not null AND category='sites' AND id in (" + "{{ str(ti.xcom_pull(key='id_list', task_ids=['hook_task'])).replace('[','').replace(']','').replace('(','').replace(')','').replace(',,','') }}" + ") group by category),0) AS TRIGGER;", . I also tried to apply Python operator within SQL variable as given below but no luck, code given below:- sql="SELECT COALESCE ( (SELECT count(*) FROM lct_transformation_history WHERE deltaload_transformation_status is not null AND fullload_transformation_status is not null AND category='sites' AND id in (" + "{{ str(ti.xcom_pull(key='id_list', task_ids=['hook_task'])).replace('[','').replace(']','').replace('(','').replace(')','').replace(',,','') }}" + ") group by category),0) AS TRIGGER;", .

Looking forward for valuable input.期待宝贵的意见。 Thanks.谢谢。

I have solved my problem with following solution, since my Xcom response was in tuple & list hence i solved through a code change given below:-我已经通过以下解决方案解决了我的问题,因为我的 Xcom 响应在元组和列表中,因此我通过下面给出的代码更改解决了:-

"{{ (ti.xcom_pull(key='id_list', task_ids=['hook_task'])).str()[0][0][0] }}

and it gave me desired result what i was expecting as 13 .它给了我期望的结果13

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM