在自定义操作员气流中拉取 xcom 值

Question

I wrote a custom operator called HadoopPutHdfs in Airflow, so I need to pass xxx parameter to HadoopPutHdfs and I need to fill xxx with the return value from the generate_file_path task

with DAG(dag_id='my_custom_operator_dag', schedule_interval='1 * * * *', default_args=default_args, catchup=False) as dag:

    generate_file_path = PythonOperator(
        task_id='generate_file_path',
        python_callable=generate_file_path_func,
        dag=dag,

    )

    put_to_hdfs = HadoopPutHdfs(
        task_id='put_to_hdfs',
        headers={'Content-Type': 'text/plain'},
        hdfs_path='webhdfs/v1/user/hive/13.zip',
        hadoop_host='10.10.10.146',
        hadoop_port=9870,
        source_path='/opt/airflow/dags/1.zip',
        dag=dag,
        xxx= "{{ ti.xcom_pull(task_ids=['generate_file_path']) }}",

    )

this line not work ,

xxx= "{{ ti.xcom_pull(task_ids=['generate_file_path']) }}"

How can I pass the amount of ‍‍‍ generate_file_path function to xxx perameter?

Answer 1

Sounds like you are missing the definition of xxx as a template_field in your custom operator. For example:

class CustomDummyOperator(BaseOperator):
    template_fields = ('msg_from_previous_task',)

    @apply_defaults
    def __init__(self,
                 msg_from_previous_task,
                 *args, **kwargs) -> None:
        super(CustomDummyOperator, self).__init__(*args, **kwargs)
        self.msg_from_previous_task = msg_from_previous_task

    def execute(self, context):
        print(f"Message: {self.msg_from_previous_task}")

DAG:

def return_a_str():
    return "string_value_from_op1"


task_1 = PythonOperator(
    task_id='task_1',
    dag=dag,
    python_callable=return_a_str,
)

task_2 = CustomDummyOperator(
    task_id='task_2',
    dag=dag,
    msg_from_previous_task="{{ ti.xcom_pull(task_ids='task_1') }}"
)

The output from task_2 is: Message: string_value_from_op1

You could use XcomArg for a cleaner syntax:

task_2 = CustomDummyOperator(
    task_id='task_2',
    dag=dag,
    msg_from_previous_task=task_1.output
    # msg_from_previous_task="{{ ti.xcom_pull(task_ids='task_1') }}"
)