Docker 容器随机关闭

Question

. Answers to this question are eligible for a +50 reputation bounty. okay wants to to this question:

I need an explanation as to why my container is randomly stopping. Been having this issue for many months

So I've had this system where I had 3 docker containers running at the same time to host my APIs, one for traefik, and the other two for different APIs (written in python using FastAPI). It was working fine for a while, however now all the docker containers seem to be randomly shutting down after a couple hours at the same.

This is the error message all of them output:

Exception in thread Thread-5:
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1277, in request
  File "http/client.py", line 1323, in _send_request
  File "http/client.py", line 1272, in endheaders
  File "http/client.py", line 1032, in _send_output
  File "http/client.py", line 972, in send
  File "docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 727, in urlopen
  File "urllib3/util/retry.py", line 410, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1277, in request
  File "http/client.py", line 1323, in _send_request
  File "http/client.py", line 1272, in endheaders
  File "http/client.py", line 1032, in _send_output
  File "http/client.py", line 972, in send
  File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 926, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "compose/cli/log_printer.py", line 168, in tail_container_logs
  File "compose/cli/log_printer.py", line 185, in wait_on_exit
  File "compose/container.py", line 268, in wait
  File "docker/utils/decorators.py", line 19, in wrapped
  File "docker/api/container.py", line 1305, in wait
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 233, in _post
  File "requests/sessions.py", line 578, in post
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Here is my docker-compose for my first API (call it "Hierarchy"):

services:

  backend:
    build: ./
    restart: always
    labels:
      - traefik.enable=true

      - traefik.http.services.hierarchy_app.loadbalancer.server.port=80

      - traefik.http.routers.hierarchy-http.entrypoints=http
      - traefik.http.routers.hierarchy-http.rule=Host(`api.mydomain.me`)
      - traefik.docker.network=traefik-public

      - traefik.http.routers.hierarchy-https.entrypoints=https
      - traefik.http.routers.hierarchy-https.rule=Host(`api.mydomain.me`)
      - traefik.http.routers.hierarchy-https.tls=true

      - traefik.http.routers.hierarchy-https.tls.certresolver=le

      - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true

      - traefik.http.routers.hierarchy-http.middlewares=https-redirect

    networks:
      - traefik-public

    volumes:
      - ${PWD}/hierarchy_app/commands.json:/hierarchy_app/commands.json:Z

networks:
  traefik-public:
    external: true

Here is my docker-compose for my second API (call it "Voicerooms"):

services:

  backend:
    build: ./
    restart: always
    labels:
      - traefik.enable=true

      - traefik.http.services.voicerooms_app.loadbalancer.server.port=80

      - traefik.http.routers.voicerooms-http.entrypoints=http
      - traefik.http.routers.voicerooms-http.rule=Host(`api.mydomain2.app`)
      - traefik.docker.network=traefik-public

      - traefik.http.routers.voicerooms-https.entrypoints=https
      - traefik.http.routers.voicerooms-https.rule=Host(`api.mydomain2.app`)
      - traefik.http.routers.voicerooms-https.tls=true

      - traefik.http.routers.voicerooms-https.tls.certresolver=le

      - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true

      - traefik.http.routers.voicerooms-https.middlewares=https-redirect
    networks:
      - traefik-public

    volumes:
      - ${PWD}/voicerooms_app/commands.json:/voicerooms_app/commands.json:Z

networks:
  traefik-public:
    external: true

Any help would be greatly appreciated, I've been trying to figure out this problem for a while:)

Edit: After scrolling up a bit more, I found this in the error message:

backend_1  | [2022-01-10 02:04:51 +0000] [1] [INFO] Handling signal: term
backend_1  | [2022-01-10 02:04:51 +0000] [7] [INFO] Shutting down
backend_1  | [2022-01-10 02:04:51 +0000] [8] [INFO] Shutting down

I'm assuming my docker container is getting terminated for some reason, what's going on here?

Answer 1

I also encountered the same error as you,My problem is that docker does not start.

check docker status

 systemctl status docker

systemctl start docker

docker-compose up try again

systemctl enable docker

to start on boot.

OR

If you're on a Mac, it may mean that Docker itself isn't running. I had rebooted my Mac and Docker wasn't set to automatically launch at login. You can set this as a Docker preference.

OR you can solve this by running sudo systemctl start docker and then running sudo docker-compose up

Answer 2

Seems your application is for some reason attempting to use a file that is not there, and it's not managing properly the Exception, making your application exit and your container stop.

Make sure that the send method you have in your application requests/adapters.py on line 498 is properly encapsulated into a Try...Except statement. Also, make sure to log the exception, this will avoid your application to crash, but will not fix the bug or underlying cause - you will have to investigate this separately.

Answer 3

Your application dies for some reason and the containers finish. How to debug this is a common question, I would start all containers in an interactive mode and check out what is happening:

docker run --rm -p8080:80 -it <CONTAINER NAME>

Instead of -p8080:80 (exposing container's port 80 on host's 8080), add your networking setup... Also add any mount/binds/environment setting your containers need.

You can also start the container in bash and start your command manually:

docker run --rm -p8080:80 -it <CONTAINER NAME> /bin/bash

From now on, you have to find what is wrong with your app.