airflow - dynamically change namespace in KubernetesPodOperator

Question

I am running airflow 1.10.13 on kubernetes and trying to find a way to dynamically change the namespace I run a task on. I tried using templates and inserting parameters from the dag_run.conf json but the template is only rendered in the 'cmds' and not in other task fields like namespace.

I would love to find a solution (using templates or any other way) to alter the namespace.

default_args = {
    'owner': 'airflow',
    'start_date': days_ago(1)
}

with DAG('test_ns', default_args=default_args, schedule_interval='@once') as dag:
    ns = """ {{ dag_run.conf.ns }} """
    example_task= KubernetesPodOperator(namespace=ns,
                                         image='python:3.6',
                                         cmds=["/bin/sh", "-c", "echo {{ns}}"],
                                         arguments=[],
                                         task_id='example_task',
                                         name='example_task',
                                         get_logs=True,
                                         is_delete_operator_pod=True,
                                         provide_context=True
                                         )

Answer 1

The list of templated fields in KubernetesPodOperator does not have namespace .

You can create your own operator with the same behavior that adds namespace to template_fields :

class MyKubernetesPodOperator(KubernetesPodOperator):
         template_fields = KubernetesPodOperator.template_fields +('namespace',)
    

Then you can use your code as:

with DAG('test_ns', default_args=default_args, schedule_interval='@once') as dag:
    ns = """ {{ dag_run.conf.ns }} """
    example_task= MyKubernetesPodOperator(namespace=ns,...)

EDIT: Full example with your code:

from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator


class MyKubernetesPodOperator(KubernetesPodOperator):
    template_fields = KubernetesPodOperator.template_fields + ('namespace',)


default_args = {
    'owner': 'elad',
    'start_date': datetime(2019, 11, 1),

}

with DAG(dag_id='stackoverflow',
         default_args=default_args,
         schedule_interval=None
         ) as dag:
    ns = """ {{ dag_run.conf.ns }} """
    example_task = MyKubernetesPodOperator(namespace=ns,
                                           image='python:3.6',
                                           cmds=["/bin/sh", "-c", "echo {{ns}}"],
                                           arguments=[],
                                           task_id='example_task',
                                           name='example_task',
                                           get_logs=True,
                                           is_delete_operator_pod=True,
                                           )

Triggered the dag with conf :

{"ns":"mynamespace"}