使用docker-compose运行分布式气流架构时如何将新用户添加到docker镜像

Question

(THE ORIGINAL QUESTION WAS EDITED TO MAKE IT MORE CLEAR)

1. SOLUTION AT THE END OF THE QUESTION
2. ANOTHER SOLUTION IN THE ANSWER

The goal and the setup

The main goal is to run container based processing(using the DockerOperator) when the airflow celery worker is also running inside a docker container. At the moment, I'm testing the setup at one machine, but in the end I'll run the celery worker containers at separate machines operating in the same network sharing some of the airflow specific mount points(dags,logs,plugins) and user ids etc.

I'm launching the whole setup from a docker-compose.yml where I set AIRFLOW_UID to match my UID at the host machine and AIRFLOW_GID to 0 as suggested in the airflow documentation. At the host, my UID belongs to docker group, but it doesn't belong to group 0. The /var/run/docker.sock is mounted into the containers.

TEST 1

I followed the example represented here https://towardsdatascience.com/using-apache-airflow-dockeroperator-with-docker-compose-57d0217c8219 . Using the above-mentioned setup with the official airflow image 2.1.4 and DockerOperator. Task run fails, which is related to the fact that the default user doesn't have the needed permissions to /var/run/docker.sock . (I still need to check if adding the user to group 0 at the host would solve the issue as pointed out by @JarekPotiuk in the his comment. The problem is that group 0 is the root group and most likely I'll not get permission to add the user to it)

[2021-09-27 05:38:30,863] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1291, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1337, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1286, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1046, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 984, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

TEST 2

I created custom image from the official image by adding 'newuser' with an UID that matches my UID at the host and 'docker' group that matches the one at the host.

However, when I launch the setup, the user I created in the image build phase is not there and I can't understand why. There is a 'default' user with uid=1234 and gid=0. This default user is created if I use the official Image and just define AIRFLOW_UID in the docker-compose.yml.

Dockerfile:

FROM apache/airflow:2.1.0

USER root
RUN useradd newuser -u 1234 -g 0

RUN groupadd --gid 986 docker \
    && usermod -aG docker newuser
USER newuser

Also, if I don't create the newuser and just add airflow user to docker group then the airflow user is really added to the docker group as it should.

Does docker-compose overwrite the users created at the image build phase? What would be the best way to solve this issue?

SOLUTION

This solution makes it possible to user DockerOperator from airflow container to launch DockerContainers at host.

You can choose either the default UID=50000 and GID=0 or a custom UID and GID=0. Create a docker group at the host and add the chosen UID to it. Then add the airflow user inside the container into the docker group. You can do this by adding the group in the compose file

group_add:
  - <docker GID>

In addition, you have to mount the docker.sock file to the container

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

and add a variable AIRFLOW__CORE__ENABLE_XCOM_PICKLING=True

Answer 1

I'm launching the whole setup from a docker-compose.yml where I set AIRFLOW_UID=1234 and AIRFLOW_GID=0. I'm using a docker image based on the official airflow image with the addition that I have created 'newuser' with gid=1234 and 'docker' group with gid that matches the one at the host.

You should not do it at all. The user will be created automatically by Airflow's image entrypoint when you use a differnt UID than default - see https://airflow.apache.org/docs/docker-stack/entrypoint.html#allowing-arbitrary-user-to-run-the-container . In fact all that you want to do should be possible without having to extend the Airflow image.

What you need to do, you need to create this user that you want to run inside the container ON THE HOST - not in the container. And it should belong to the docker group ON THE HOST - not in the container.

Docker works in the way that it uses the same kernel/users that are defined in the system, so when you run something as a user in the container, it is run with the "host" user priviledges, so you you map your docker socket to within the container, it will be able to use the socket/run docker command becaue it will have the right permissions on the host.

Therefore (in case you run your docker-compose as regular user who already belongs to docker group) the best way is the one suggested in the quick-start - ie run airflow with your "host" user that you are logged in with: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

This also makes all the files created in container belong to the "logged in user" (if they are created in directories mounted inside - such as logs directory).

But if your goal is to use it in "unattended" environment, then likely creating the new user on your host and adding the user to both 0 and docker groups should solve the problem.

Answer 2

As a complement to the great answer from @JarekPotiuk, if as indicated in your comments the problem is related to permissions issues when using the DockerOperator , you can try the following approach.

The idea is including in the airflow docker-compose.yml file a service based on the bobrik/socat image. Something like:

docker-proxy:
  image: bobrik/socat
  command: "TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock"
  ports:
    - 2375:2375
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  restart: always

This will effectively create a bridge with you host docker daemon and would allow you to run your containers using the DockerOperator without permissions issues by providing an appropriate value for the docker_url argument:

docker_based_task = DockerOperator(
    task_id="a_docker_based_one",
    docker_url="tcp://docker-proxy:2375"
    # ...
)