简体繁体 English

如何设置气流 DAG 连续运行

[英]How to set airflow DAG runs continuously

原文 2017-11-21 20:46:07 4 2 python/ airflow/ apache-airflow

I am using airflow to schedule a spark job, I need to got a DAG "run continuously" -- Like I just want a DAG to run, and when it finishes, to start a new DAG instance again.我正在使用气流来安排一个火花作业，我需要让 DAG“连续运行”——就像我只想运行一个 DAG，当它完成时，再次启动一个新的 DAG 实例。

I have two options in my mind:我脑子里有两个选择：
1. Allow only one DAG instance running at a time and run the DAG more frequently 1. 一次只允许运行一个 DAG 实例，并且更频繁地运行 DAG
2. Have another DAG to watch and kick off when it is needed. 2. 有另一个 DAG 来观察并在需要时启动。

Have anyone implement it?有人实施吗？ Suggestions?建议？ Thanks!谢谢！

2 个解决方案

Schedule a continually airflow DAG run 安排持续气流 DAG 运行

You should be able to use the TriggerDagRunOperator to rerun the DAG after the last task is finished.您应该能够在最后一个任务完成后使用 TriggerDagRunOperator 重新运行 DAG。

Why do you need airflow to do that?为什么需要气流来做到这一点？ You could implement an infinite loop in your spark job.您可以在 Spark 作业中实现无限循环。 It seems more efficient.看起来效率更高。

如何防止气流回填 dag 运行？ - How to prevent airflow from backfilling dag runs?

如何获取气流 dag 运行的 JobID？ - How to get the JobID for the airflow dag runs?

如何在气流中设置多个Dag目录 - How to set up multiple Dag directories in airflow

气流回填和新的dag运行 - Airflow backfills and new dag runs

Airflow 不仅为 DAG 间隔系列的最新实例运行 DAG Run，而且在我设置 catchup=False 时也运行它之前的实例 - Airflow not only runs a DAG Run for the most current instance of the DAG interval series but also the one before it when I set catchup=False

将 Airflow 触发的 DAG 标记为失败/成功会影响 DAG 的所有运行 - Mark Failed/Success for Airflow triggered DAG effects all runs of the DAG

Airflow - BashOperator：获取“Dag 运行已为 DAG 死锁：...”错误 - Airflow - BashOperator: Getting “Dag runs are deadlocked for DAG:..” error

如何控制气流中的DAG并发性 - How control DAG concurrency in airflow

如何停止预定的气流 - how to stop airflow scheduled dag

如何在循环中实施气流DAG - how to implement airflow DAG in a loop

暂无

暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 如何防止气流回填 dag 运行？ - How to prevent airflow from backfilling dag runs? 如何获取气流 dag 运行的 JobID？ - How to get the JobID for the airflow dag runs? 如何在气流中设置多个Dag目录 - How to set up multiple Dag directories in airflow 气流回填和新的dag运行 - Airflow backfills and new dag runs Airflow 不仅为 DAG 间隔系列的最新实例运行 DAG Run，而且在我设置 catchup=False 时也运行它之前的实例 - Airflow not only runs a DAG Run for the most current instance of the DAG interval series but also the one before it when I set catchup=False 将 Airflow 触发的 DAG 标记为失败/成功会影响 DAG 的所有运行 - Mark Failed/Success for Airflow triggered DAG effects all runs of the DAG Airflow - BashOperator：获取“Dag 运行已为 DAG 死锁：...”错误 - Airflow - BashOperator: Getting “Dag runs are deadlocked for DAG:..” error 如何控制气流中的DAG并发性 - How control DAG concurrency in airflow 如何停止预定的气流 - how to stop airflow scheduled dag 如何在循环中实施气流DAG - how to implement airflow DAG in a loop

相关标签

粤ICP备18138465号 © 2020-2024 STACKOOM.COM