将 dag_run_id 值从一个任务传递到另一个 airflow 任务

Question

def my_function(**kwargs):
    global dag_run_id
    dag_run_id = kwargs['dag_run'].run_id

example_task = PythonOperator(
    task_id='example_task',
    python_callable=my_function,
    provide_context=True,
    dag=dag)

bash_task = BashOperator(
        task_id = 'bash_task' ,
        bash_command = 'echo {{ dag_run_id }}' ,
        dag = dag ,
        )

bash_task is not printing value of dag_run_id and i want to use dag_run_id in other tasks

Answer 1

XCom (Cross-Communication) is the mechanism in Airflow that allows you to share data between tasks. Returning a value from a PythonOperator's callable automatically stores the value as an XCom. So your Python function could do:

def my_function(**kwargs):
    dag_run_id = kwargs["run_id"]
    return dag_run_id

Note that run_id is one of the templated variables given by Airflow, see the full list here: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#variables .

This stores the returned value as an "XCom" in Airflow. You can observe XComs via the Grid View -> select task -> XCom, or see all XCom values via Admin -> XComs. The task-specific XCom view shows something like this:

You can then fetch (known as "pull" in Airflow) the value in another task:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command="echo {{ ti.xcom_pull(task_ids='example_task') }}",
)

This will fetch the XCom value from the task with id example_task and echo it.

The full DAG code looks like this:

import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator

with DAG(
    dag_id="so_75213078",
    start_date=datetime.datetime(2023, 1, 1),
    schedule_interval=None,
):

    def my_function(**kwargs):
        dag_run_id = kwargs["run_id"]
        return dag_run_id

    example_task = PythonOperator(task_id="example_task", python_callable=my_function)

    bash_task = BashOperator(
        task_id="bash_task",
        bash_command="echo {{ ti.xcom_pull(task_ids='example_task') }}",
    )

    example_task >> bash_task

Tasks are executed by separate processes in Airflow (and sometimes on separate machines), therefore you cannot rely on eg global or a local file path to exist for all tasks.

Answer 2

To pass a variable value from a task to another one, you can use Airflow Cross-Communication (XCom) as explained in the other answer.

But if you just want to pass the dag_run id, you don't need to do all of this, where it's available on all the tasks:

bash_task = BashOperator(
    task_id = 'bash_task' ,
    bash_command = 'echo {{ dag_run.run_id }}' ,
    dag = dag ,
)