[英]Calling a BashOperator from a Python Callable with XCom Pull in Airflow Dag
I am attempting to return a list of values from an airflow task, and then within another task loop over the list and call a BashOperator using the value as an argument to a python script.我试图从气流任务返回一个值列表,然后在另一个任务中循环遍历列表并使用该值作为 python 脚本的参数调用 BashOperator。
If using a PythonOperator is the correct way then I would be intersted in seeing how to do that, but I do want to have the Python script I am calling be an external file and not another callable in my Airflow Dag.如果使用 PythonOperator 是正确的方法,那么我会很想知道如何做到这一点,但我确实希望我调用的 Python 脚本是一个外部文件,而不是我的 Airflow Dag 中的另一个可调用文件。
I am able to loop over the list and print out the value, however I can't figure out how to call the BashOperator which I have placed in the loop.我能够遍历列表并打印出值,但是我不知道如何调用我放置在循环中的 BashOperator。
Here is a simple version of the code, just attempting to echo the value:这是代码的一个简单版本,只是尝试回显值:
from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
DAG = DAG(
dag_id='locations_test',
start_date=datetime.now(),
schedule_interval='@once'
)
def get_locations_function(**kwargs):
locations = ['one','two','three']
return locations
get_locations_task = PythonOperator(
task_id='get_locations_task',
python_callable=get_locations_function,
provide_context=True,
dag=DAG)
def call_loop_over_locations_function(**kwargs):
ti = kwargs['ti']
locations = ti.xcom_pull(task_ids='get_locations_task')
for loc in locations:
print(loc) #this prints
bash_operator = BashOperator(
task_id='do_things_with_location',
bash_command="echo '%s'" %loc,
xcom_push=True,
dag=DAG)
#this doesn't get called, i have also tried
#call_loop_over_locations_task >> bash_operator
bash_operator
call_loop_over_locations_task = PythonOperator(
task_id='call_loop_over_locations_task',
python_callable=call_loop_over_locations_function,
provide_context=True,
dag=DAG)
get_locations_task >> call_loop_over_locations_task
**kwargs
in your call_loop_over_locations_function()
is actually a context. **kwargs
你call_loop_over_locations_function()
**kwargs
实际上是一个上下文。 You can use it to execute bash operator.您可以使用它来执行 bash 操作符。
code代码
from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
DAG = DAG(
dag_id='locations_test',
start_date=datetime(2020, 1, 1),
schedule_interval=None
)
def get_locations_function(**kwargs):
locations = ['one','two','three']
return locations
get_locations_task = PythonOperator(
task_id='get_locations_task',
python_callable=get_locations_function,
provide_context=True,
dag=DAG)
def call_loop_over_locations_function(**kwargs):
print(kwargs)
ti = kwargs.get('ti')
locations = ti.xcom_pull(task_ids='get_locations_task')
for loc in locations:
print(loc) #this prints
bash_operator = BashOperator(
task_id='do_things_with_location',
bash_command="echo '%s'" %loc,
xcom_push=True,
dag=DAG)
bash_operator.execute(context=kwargs)
#this doesn't get called, i have also tried
#call_loop_over_locations_task >> bash_operator
bash_operator
call_loop_over_locations_task = PythonOperator(
task_id='call_loop_over_locations_task',
python_callable=call_loop_over_locations_function,
provide_context=True,
dag=DAG)
get_locations_task >> call_loop_over_locations_task
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.