[英]Trigger airflow DAG manually with parameter and pass then into python function
I want to pass parameters into airflow DAG and use them in python function.我想将参数传递到 airflow DAG 并在 python function 中使用它们。 I can use the parameter into bash operator, but I can't find any reference to use them as python function.
我可以将参数用于 bash 运算符,但我找不到任何将它们用作 python function 的参考。
from airflow import DAG
from airflow.operators.bash_operator import BashOperator from airflow.operators.python_operator import PythonOperator from airflow.utils.dates import days_ago
#Define DAG
dag = DAG("test_backup", schedule_interval=None, start_date=days_ago(1))
#Parameter
owner="{{ dag_run.conf['owner'] }}"
table="{{ dag_run.conf['table'] }}"
run_this="echo "+owner+"."+table
def test_func(owner,table):
print(owner+"."+table)
task1 = BashOperator(
task_id='test_task1',
bash_command=run_this,
dag=dag,
queue='cdp_node53',
)
task2 = PythonOperator(
task_id='test_task2',
python_callable=test_func(owner,table),
dag=dag,
queue='cdp_node53',
)
I want to pass below as parameters while trigger DAG.我想在触发 DAG 时将以下参数作为参数传递。 "task1" works fine for me.
“task1”对我来说很好。 I need to make "task2" workable.
我需要使“task2”可行。 Please guide me to correct the above code so that I can pass parameters into it.
请指导我更正上面的代码,以便我可以将参数传递给它。
{"owner":"test_owner","table":"test_table"}
For passing arguments into the PythonOperator
you should use either op_args
(for positional arguments) or op_kwargs
(for keyword arguments).要将 arguments 传递给
PythonOperator
,您应该使用op_args
(用于位置参数)或op_kwargs
(用于关键字参数)。 Both parameters are also template fields so the values can be Jinja expressions as well.这两个参数也是模板字段,因此值也可以是 Jinja 表达式。
Refactoring your code using op_kwargs
:使用
op_kwargs
重构代码:
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
#Define DAG
dag = DAG("test_backup", schedule_interval=None, start_date=days_ago(1))
#Parameter
owner="{{ dag_run.conf['owner'] }}"
table="{{ dag_run.conf['table'] }}"
run_this="echo "+owner+"."+table
def test_func(owner,table):
print(owner+"."+table)
task1 = BashOperator(
task_id='test_task1',
bash_command=run_this,
dag=dag,
queue='cdp_node53',
)
task2 = PythonOperator(
task_id='test_task2',
python_callable=test_func,
op_kwargs={"owner": owner, "table": table},
dag=dag,
queue='cdp_node53',
)
Both tasks will log the INFO - test_owner.test_table
now.这两个任务现在都将记录
INFO - test_owner.test_table
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.