Gunicorn is stuck in deadlock for python django service running in docker container

Question

I am running a python django webhook application that runs via gunicorn server. My setup is nginx + Gunicorn + Django. Here is what the config looks like: gunicorn app1.wsgi:application --bind 0.0.0.0:8000 --timeout=0

The application runs perfectly for ~1 -2 million request, but after running for few hours the gunicorn shows in sleep state and then no more webhook event gets received root 3219 1.3 0.0 256620 61532? Sl 14:04 0:19 /usr/local/bin/python /usr/local/bin/gunicorn app1.wsgi:application --bind 0.0.0.0:8000 --timeout=0 root 3219 1.3 0.0 256620 61532? Sl 14:04 0:19 /usr/local/bin/python /usr/local/bin/gunicorn app1.wsgi:application --bind 0.0.0.0:8000 --timeout=0

The service is running in 4 different containers and within few hours this behavior is observed for 1 container and then it occurs for one or more containers in subsequent hours.

I tried sending a signal to reload gunicorn config which is able to bring the gunicorn process into running state.

What is curious is that when I run 4 django containers, for few requests it works perfectly well. But continuously receiving traffic causes this deadlock in one of the gunicorn worker's state and it keep on waiting for a trigger to start accepting traffic again while rest of the three gunicorn workers are healthy and running !

Question - Why does gunicorn worker process is going in sleep state(Sl)? How can this behavior be fixed?

Answer 1

Try to put Nginx as reverse-proxy in front of gunicorn. Perhaps gunicorn incorrectly handles invalid clients which results in deadlock. Nginx by itself provides buffering which saves gunicorn from bad things like eternal opened TCP connections.

See gunicorn documentation about proper deployment - https://docs.gunicorn.org/en/stable/deploy.html

See this issue for problem discussion - https://github.com/benoitc/gunicorn/issues/2917