Using apache airflow docker operator with rootless docker

Question

I'm working on a project that is using apache airflow to schedule different scripts.

Airflow itself and the scripts setting up the DAGs are bundled up in one Dockerfile. The central part of each DAG is a dockeroperator which starts the appropiate container on the host system.
(Note: It is a requirement that each airflow and the scripts its schedules each have a Dockerfile)

Currently docker on the host system has root privileges as suggested here . When starting up my airflow container I create a volume with -v /var/run/docker.sock:/var/run/docker.sock . In this configuration airflow starts the containers as expected and without problems.

However because running docker with root privileges seems unsafe I tried the rootless mode for docker ( https://docs.docker.com/engine/security/rootless/ ).

After the setup the all containers individually work perfectly fine on the host machine. Only when I run the airflow container do I get issues.

The error message is quite long, so I'll just post the last line: docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

This error is the same one I got while I was not creating the volume mentioned above in the earlier iteration of my whole setup, so the issue is probably related to the docker daemon in the airflow container not having the correct premissions, but I don't know how to fix this.

Answer 1

You could launch the whole setup using docker compose. A good starting point for the docker-compose file is the one given here: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

You can choose either the default UID=50000 and GID=0 or a custom UID. Create this UID at host, create a docker group and add this user to the group. Then add the airflow user inside the container into this group. You can do this at compose file by adding

group_add:
  - <docker GID>

Note. If you use a custom image and add the user to docker group in the Dockerfile it has no effect since these settings are overwritten by docker compose startup.sh

In addition, you have to mount the docker.sock file to the container

volumes:
  - /var/run/docker.sock:/var/run/docker.sock