How to mount folder/file to KubernetesPodOperator Airflow

Question

I need to run a python script from a KubernetesPodOperator, so I want to mount the python file into the Python docker Image. Reading some posts

• How to mount volume of airflow worker to airflow kubernetes pod operator?
• Mounting volume issue through KubernetesPodOperator in GKE airflow
• Mounting folders with KubernetesPodOperator on Google Composer/Airflow
• https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py#L107
• https://www.aylakhan.tech/?p=655

it doesn't get clear at all for me.

The python file is located in the route /opt/airflow/dags/test_dag , so I would like to mount the entire folder and not only the script. I have tried with:

    vol1 = k8s.V1VolumeMount(
        name='test_volume', mount_path='/opt/airflow/dags/test_dag'
    )
    volume = k8s.V1Volume(
        name='test-volume',
        persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='test-volume'),
    )

    k = KubernetesPodOperator(
        task_id="dry_run_demo",
        cluster_name="eks",
        namespace="data",
        image="python:3.9-buster",
        volumes=[volume],
        volume_mounts=[vol1],
        arguments=["echo", "10"],
    )

But I am getting the error:

Pod "pod.388baaaa7c27489c9dd5f7f37ee8ce5b" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "test_volume\

I am using Airflow 2.1.1 deployed in a EC2 with docker-compose and apache-airflow-providers-cncf-kubernetes==3.0.1

Answer 1

The error means that the names don't match. you defined name='test_volume' for V1VolumeMount and name='test-volume for V1Volume .

To solve your issue names should be identical.

vol1 = k8s.V1VolumeMount(
    name='test-volume', mount_path='/opt/airflow/dags/test_dag'
)
volume = k8s.V1Volume(
    name='test-volume',
    persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='test-volume'),
)