[英]Dynamically created tasks/dags are not working in apache airflow
I'm created a DAG with scheduling @daily interval and separate task id for every workflow. 我创建了一个DAG,其中安排了@daily间隔并为每个工作流分配了单独的任务ID。 But Its is not running as excepted. 但是它没有按例外运行。 Is it possible to do like this? 有可能这样做吗? Is there any other ways to create dynamic tasks for a particular dag? 还有其他方法可以为特定的dag创建动态任务吗? And to pause a particular task instance using command line? 并使用命令行暂停特定的任务实例?
from __future__ import print_function
from builtins import range
from airflow.operators import PythonOperator,DummyOperator,BranchPythonOperator,SqlSensor
from airflow.models import DAG
from datetime import datetime, timedelta
import time
from pprint import pprint
seven_days_ago = datetime.combine(
datetime.today() - timedelta(7), datetime.min.time())
args = {
'owner': 'varakumar',
'start_date': seven_days_ago,
}
dag = DAG(
dag_id='dynamic_task_creation', default_args=args,
schedule_interval="@daily")
def get_decision():
return "right"
start = DummyOperator(
task_id='start',
dag=dag)
td=datetime.today()
x=str(datetime(td.year,td.month,td.day,td.hour,td.minute,td.second)).replace (" ", "_").replace (":", "-")
pause_task_id = ("pause-%s" % x)
pause = DummyOperator(
task_id=pause_task_id,
dag=dag)
pause.set_upstream(start)
decision = BranchPythonOperator(
task_id='decision',
python_callable=lambda: get_decision(),
dag=dag)
decision.set_upstream(pause)
left = DummyOperator(
task_id='left',
dag=dag)
left.set_upstream(decision)
right = DummyOperator(
task_id='right',
dag=dag)
right.set_upstream(decision)
Thank you in advance 先感谢您
The first problem I see is that you're using a dynamic start_date
. 我看到的第一个问题是您使用的是动态start_date
。 I've seen some weird behavior when doing that, and I think it's based on the way that airflow maintains its list of past dagruns. 我在执行此操作时看到了一些奇怪的行为,并且我认为它是基于气流保持其过去的数据的方式。 Try to specify a fixed start_date
and see if that resolves anything. 尝试指定一个固定的start_date
,看看是否可以解决任何问题。
In either case, the airflow docs advise against dynamic start dates (scroll down a bit and read the description for start_date
). 无论哪种情况,气流文档都建议不要使用动态开始日期 (向下滚动并阅读start_date
的描述)。
EDIT: Also check this out for more info on that. 编辑:另外,请查阅该出来对更多的信息。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.