简体   繁体   English

如何使用 python 运算符在气流 dag 文件中使用 xCom?

[英]How to use xCom in airflow dag file using python operator?

I'm working on this airflow dag file to do some test with XCOM, but not sure how to use it between python operators.我正在处理这个气流 dag 文件以对 XCOM 进行一些测试,但不确定如何在 python 运算符之间使用它。 Can someone please help how to write the logic to pass a message between the python operators using XCOM push and pull functions.有人可以帮助如何编写逻辑以使用 XCOM 推拉功能在 python 运算符之间传递消息。 Below is the dag file that i'm working on...下面是我正在处理的 dag 文件...

The question is how to pass a message from each task to another task问题是如何将消息从每个任务传递到另一个任务

@dag_factory
def create_dag():
    with DAG(
        dag_id="DSStest",
        default_args=default_args,
        schedule_interval=timedelta(1),
    ) as dag:
        # Define operators here, for example:

        output_file = path_in_workspace("testout")

        rscript_file = path_in_workspace("rtest2.R ")

        bcmd1 = "downloading some file here..."

        t0 = PythonOperator(
            task_id="start",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 0],
        )

        t1 = PythonOperator(
            task_id="job1",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 1],
        )

        t2 = PythonOperator(
            task_id="job2",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 2],
        )

        t10 = PythonOperator(
            task_id="job10",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 10],
        )

        t20 = BashOperator(
            task_id="job20",
            bash_command=bcmd1,
            queue={
                "worker_type": "png-bash-worker",
                "request_memory": "1G",
                "request_cpu": 1,
            },
        )

        # Define dependencies between operators here, for example:

        t0 >> t1
        t0 >> t2
        t1 >> t10
        t2 >> t10
        t10 >> t11
        t11 >> t20

        return dag  # Do not change this

I recommend to take a look at this example, it shows/explains everything regarding xcoms and PythonOperators:我建议看一下这个例子,它显示/解释了关于 xcoms 和 PythonOperators 的一切:
example_xcom.py example_xcom.py

The official airflow page also explains xcoms in a lot of detail:官方的airflow页面也对xcoms做了很多详细的解释:
official documentation官方文件

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 使用 XCom Pull in Airflow Dag 从 Python Callable 调用 BashOperator - Calling a BashOperator from a Python Callable with XCom Pull in Airflow Dag Airflow DAG:如何使用 Python 运算符而不是 BigQuery 运算符将数据插入表中? - Airflow DAG: How to insert data into a table using Python operator, not BigQuery operator? Apache Airflow 如何将 xcom_pull() 值转换为 DAG? - Apache Airflow How to xcom_pull() value into a DAG? 如何在 Airflow 运算符中应用 Python 函数超过 Airflow XCom_pull 值 - How to apply Python Functions over Airflow XCom_pull value within Airflow Operator 气流:如何将 xcom 从父 dag 传递到 subdag - Airflow: How to pass xcom from parent dag to subdag 如何使用 Python 在 Airflow 中的另一个 DAG 成功时触发 DAG? - How to Trigger a DAG on the success of a another DAG in Airflow using Python? 如何使用分支运算符在 Airflow DAG 中分支多条路径? - How to branch multiple paths in Airflow DAG using branch operator? 将Python文件转换为Airflow DAG - Converting Python file to Airflow DAG Airflow Python Operator - 从 xcom pull 获取参数值 - Airflow Python Operator - Get values from xcom pull for params 如何自动将所有 DAG 文件移动到 Airflow Docker 而不仅仅是将最新添加的文件移动并重命名为“example_python_operator”? - How to automatically move all DAG files to Airflow Docker and not just have only the latest added file moved and renamed to 'example_python_operator'?
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM