How can I make the Docker operator work in Airflow?

Question

I somehow can not make the Docker Operator work in Airflow.

The task looks like this:

DockerOperator( task_id = 'Start_Database',
                               image='name_of_the_image',
                               api_version='auto',
                               auto_remove=True,
                               docker_url='unix://var/run/docker.sock',
                               network_mode='bridge',
                               dag=dag

The dag starts but terminates after 5 minutes. When I look in the Docker UI the Image never gets executed via the task. Do I somehow have to define anything else.

Answer 1

Probably it's a permissions problem, where the user which you used to run the scheduler (or the workers if you have a celery executor), doesn't have enough permissions to use the docker socket. You can try to run airflow with root (just for debugging), if it works, you can add the user you use usually to run airflow to the docker group:

sudo usermod -aG docker <USER>