如何使用 xcom 在 airflow SSHOperator 任务之间传递 output 数据？

Question

Problem summary:

• I need to get stdout from one SSHOperator using xcom
• Filter some rows and get output values for passing them to another SSHOperator

Unfortunatelly I've not find anything helpful in the Airflow documentation

Code example:

import airflow
from airflow.operators.dummy_operator import DummyOperator
from airflow.contrib.operators.ssh_operator import SSHOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.datetime(2020, 1, 1, 0, 0),
}

dag = airflow.DAG(
    'example',
    default_args=default_args,
)

task_dummy = DummyOperator(
    task_id='task_dummy',
    dag=dag
)

cmd_ssh = """
for f in "file1" "file2"
do
    if $(hdfs dfs -test -d /data/$f)
        then hdfs dfs -rm -r -skipTrash /data/$f
        else echo "doesn't exists"
    fi
done
"""

task_1 = SSHOperator(
    ssh_conn_id='server_connection',
    task_id='task_ssh',
    command=cmd_ssh,
    do_xcom_push=True,
    dag=dag
)

My question is - how to access stdout from task_1 when I sed do_xcom_push=True?

Answer 1

You can access the XCom data in templated fields or callables which receive the Airflow context, such as the PythonOperator ( and its child classes ) -- from the documentation:

# inside a PythonOperator called 'pushing_task'
def push_function():
    return value

# inside another PythonOperator where provide_context=True
def pull_function(**context):
    value = context['task_instance'].xcom_pull(task_ids='pushing_task')