[英]Custom Operator XCom during run time in Airflow
I understand for PythonOperator/BashOperator
we can use Xcom to communicate.我了解
PythonOperator/BashOperator
我们可以使用 Xcom 进行通信。
eg例如
def func(**context):
context['task_instance'].xcom_pull()
However, I am wondering how to access xcom
for a custom operator during run time.但是,我想知道如何在运行时访问自定义运算符的
xcom
。
My operator looks like this:我的操作员看起来像这样:
class ECHOXOperator(BaseOperator):
@apply_defaults
def __init__(self, x, *args, **kwargs):
self.x = x
super(ECHOXOperator, self).__init__(*args, **kwargs)
def execute(self, context):
print(self.x)
So in my DAG:所以在我的 DAG 中:
I can do我可以
task2 = ECHOXOperator(x = 'Hello")
And it works well.它运作良好。 But how can I access x from an upstream task?
但是如何从上游任务访问 x 呢?
Something like:就像是:
def task1(**context):
task_instance = context['task_instance']
task_instance.xcom_push(key="x", value="Hello")
generate_data = PythonOperator(
task_id="task1",
python_callable=task1,
dag=dag,
)
task2 = ECHOXOperator(x = task_instance.xcom_pull('task1', 'x'), provide_context=True)
task1 >> task2
This is not working because task_instance
in ECHOXOperator
is not defined.这不起作用,因为
task_instance
中的ECHOXOperator
未定义。
Thanks谢谢
You should pass x
as templated_fields in your Custom Operator.您应该在自定义运算符中将
x
作为templated_fields传递。
class ECHOXOperator(BaseOperator):
template_fields = ['x']
@apply_defaults
def __init__(self, x, *args, **kwargs):
self.x = x
super(ECHOXOperator, self).__init__(*args, **kwargs)
def execute(self, context):
print(self.x)
And now you can do the following to get the value of Xcom passed in previous task:现在您可以执行以下操作来获取上一个任务中传递的 Xcom 的值:
def task1(**context):
task_instance = context['task_instance']
task_instance.xcom_push(key="x", value="Hello")
generate_data = PythonOperator(
task_id="task1",
python_callable=task1,
dag=dag,
)
task2 = ECHOXOperator(x = "{{ ti.xcom_pull('task1', 'x') }}")
task1 >> task2
More information regarding templated_fields and Jinja templating: https://airflow.readthedocs.io/en/latest/concepts.html#id1有关templated_fields和 Jinja 模板的更多信息: https : //airflow.readthedocs.io/en/latest/concepts.html#id1
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.