[英]airflow - dynamically change namespace in KubernetesPodOperator
I am running airflow 1.10.13 on kubernetes and trying to find a way to dynamically change the namespace I run a task on.我在 kubernetes 上运行 airflow 1.10.13 并试图找到一种方法来动态更改我运行任务的命名空间。 I tried using templates and inserting parameters from the dag_run.conf json but the template is only rendered in the 'cmds' and not in other task fields like namespace.我尝试使用模板并从 dag_run.conf json 中插入参数,但模板仅在“cmds”中呈现,而不在命名空间等其他任务字段中呈现。
I would love to find a solution (using templates or any other way) to alter the namespace.我很想找到一个解决方案(使用模板或任何其他方式)来改变命名空间。
default_args = {
'owner': 'airflow',
'start_date': days_ago(1)
}
with DAG('test_ns', default_args=default_args, schedule_interval='@once') as dag:
ns = """ {{ dag_run.conf.ns }} """
example_task= KubernetesPodOperator(namespace=ns,
image='python:3.6',
cmds=["/bin/sh", "-c", "echo {{ns}}"],
arguments=[],
task_id='example_task',
name='example_task',
get_logs=True,
is_delete_operator_pod=True,
provide_context=True
)
The list of templated fields in KubernetesPodOperator does not have namespace
. KubernetesPodOperator 中的模板化字段列表没有namespace
。
You can create your own operator with the same behavior that adds namespace
to template_fields
:您可以创建自己的运算符,其行为与将namespace
添加到template_fields
的行为相同:
class MyKubernetesPodOperator(KubernetesPodOperator):
template_fields = KubernetesPodOperator.template_fields +('namespace',)
Then you can use your code as:然后你可以使用你的代码:
with DAG('test_ns', default_args=default_args, schedule_interval='@once') as dag:
ns = """ {{ dag_run.conf.ns }} """
example_task= MyKubernetesPodOperator(namespace=ns,...)
EDIT: Full example with your code:编辑:您的代码的完整示例:
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
class MyKubernetesPodOperator(KubernetesPodOperator):
template_fields = KubernetesPodOperator.template_fields + ('namespace',)
default_args = {
'owner': 'elad',
'start_date': datetime(2019, 11, 1),
}
with DAG(dag_id='stackoverflow',
default_args=default_args,
schedule_interval=None
) as dag:
ns = """ {{ dag_run.conf.ns }} """
example_task = MyKubernetesPodOperator(namespace=ns,
image='python:3.6',
cmds=["/bin/sh", "-c", "echo {{ns}}"],
arguments=[],
task_id='example_task',
name='example_task',
get_logs=True,
is_delete_operator_pod=True,
)
Triggered the dag with conf
:用conf
触发 dag:
{"ns":"mynamespace"}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.