[英]Dynamic task generation in an Airflow DAG
I want to use Airflow to generate client reports, I would like to have one DAG that loops through all clients and launches a task to generate their report. 我想使用Airflow生成客户端报告,我希望有一个DAG循环遍历所有客户端并启动任务来生成他们的报告。 The list of clients is gathered by the first task in the DAG and cannot be hardcoded in.
客户端列表由DAG中的第一个任务收集,无法进行硬编码。
Basically I have a task that uses a database hook in a python operator to generate a list. 基本上我有一个任务,在python运算符中使用数据库钩子来生成列表。 Then for each item in the list I would like to execute a task using a python operator with that item being passed as an argument to the python function.
然后对于列表中的每个项目,我想使用python运算符执行任务,该项目作为参数传递给python函数。 Is there a certain pattern I can use to achieve this?
我可以使用某种模式来实现这一目标吗?
Trigger_dag concept Trigger_dag概念
Let the task that uses a database hook in a python operator to generate a list" be the task in the controller dag and pass the each item in list to the trigger_dag in the params section. 让在python运算符中使用数据库钩子的任务生成列表“是控制器dag中的任务,并将列表中的每个项目传递给params部分中的trigger_dag 。
You will find reference in the examples folder in your airflow installation 您可以在气流安装的examples文件夹中找到参考
Good Luck! 祝好运!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.