Django + Nginx + UWSGI 资源不可用错误 - 在 uwsgi.ini 中使用 max-request = 500 后

Question

To overcome the memory usage of the server, we have introduced max-requests = 500 to the uswgi.ini file. We face side effects like below

10.115.21.68 - - [01/Jan/2020:04:09:50 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:09:50 [error] 30522#0: *22690 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:09:48 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:09:48 [error] 30523#0: *22688 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.21.68 - - [01/Jan/2020:04:09:06 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:09:06 [error] 30522#0: *22682 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:08:49 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:08:49 [error] 30523#0: *22680 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.21.68 - - [01/Jan/2020:04:08:23 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:08:23 [error] 30522#0: *22676 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:07:51 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:07:51 [error] 30522#0: *22672 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.21.68 - - [01/Jan/2020:04:07:49 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:07:49 [error] 30522#0: *22670 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.21.68, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"
10.115.13.75 - - [01/Jan/2020:04:07:07 +0000] "GET / HTTP/1.1" 502 157 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" "-"
2020/01/01 04:07:07 [error] 30522#0: *22664 connect() to unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.115.13.75, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:"

Following is the current configuration file that I am using on my server (basically configured in AWS Cloudformation template).

  files:
    /home/uwsgi/.aws/config:
      mode: "000444"
    /srv/uwsgi.ini:
      content: !Sub |
        [uwsgi]
        socket = /tmp/uwsgi.sock
        chmod-socket=777
        chdir = /srv/app/${AppDirectory}
        pythonpath = .:..:../..
        module = ${AppName}.wsgi
        env = PYTHONPATH=.:..:../..
        enable-threads = true
        single-interpreter = true
        max-requests = 500
        processes = 4
        threads = 2
        uid = uwsgi
        gid = uwsgi
        harikari = 7200
        master = true
        harikari-verbose = true
        buffer-size = 8192
        logformat = "uwsgi:request %(addr) | %(method) %(uri) | %(status) | %(msecs) | %(ctime) | %(vszM)"
        daemonize = /var/log/uwsgi/uwsgi.log
        die-on-term = true
        for-readline = /srv/env_vars
          env = %(_)
        endfor =

    /etc/init.d/uwsgi:
      mode: '000555'
      content: |
        #!/usr/bin/env bash
        # uwsgi - this script starts and stops the uwsgi daemon
        #
        # chkconfig:   - 85 15
        # description: uwsgi serves python web apps
        # processname: uwsgi
        # pidfile:     /var/run/uwsgi/uwsgi.pid
        #

        set -e


        PATH=/sbin:/bin:/usr/sbin:/usr/bin
        DAEMON=/usr/local/bin/uwsgi
        RUN_DIR=/var/run/uwsgi
        LOG_DIR=/var/log/uwsgi
        # VASSALS_DIR=/etc/uwsgi/vassals
        NAME=uwsgi
        DESC=uwsgi
        OWNER=uwsgi
        GROUP=uwsgi

        [[ -x $DAEMON ]] || exit 0
        [[ -d $RUN_DIR ]] || mkdir $RUN_DIR && chown $OWNER:$GROUP $RUN_DIR
        [[ -d $LOG_DIR ]] || mkdir $LOG_DIR && chown $OWNER:$GROUP $LOG_DIR

        do_pid_check()
        {
            local PIDFILE=$1
            [[ -f $PIDFILE ]] || return 0
            local PID=$(cat $PIDFILE)
            for p in $(pgrep $NAME); do
                [[ $p == $PID ]] && return 1
            done
            return 0
        }


        do_start()
        {
            local PIDFILE=$RUN_DIR/$NAME.pid
            if do_pid_check $PIDFILE; then
                $DAEMON /srv/uwsgi.ini
            else
                echo "Already running!"
            fi
        }

        send_sig()
        {
            local PIDFILE=$RUN_DIR/$NAME.pid
            set +e
            [[ -f $PIDFILE ]] && kill $1 $(cat $PIDFILE) > /dev/null 2>&1
            set -e
        }

        wait_and_clean_pidfile()
        {
            local PIDFILE=$RUN_DIR/$NAME.pid
            until do_pid_check $PIDFILE; do
                echo -n "";
            done
            rm -f $PIDFILE
        }

        do_stop()
        {
            send_sig -3
            wait_and_clean_pidfile
        }

        do_reload()
        {
            send_sig -1
        }

        do_force_reload()
        {
            send_sig -15
        }

        get_status()
        {
            send_sig -10
        }

        case "$1" in
            start)
                echo "Starting $DESC: "
                do_start
                echo "$NAME."
                ;;
            stop)
                echo -n "Stopping $DESC: "
                do_stop
                echo "$NAME."
                ;;
            reload)
                echo -n "Reloading $DESC: "
                do_reload
                echo "$NAME."
                ;;
            force-reload)
                echo -n "Force-reloading $DESC: "
                do_force_reload
                echo "$NAME."
               ;;
            restart)
                echo  "Restarting $DESC: "
                do_stop
                sleep 1
                do_start
                echo "$NAME."
                ;;
            status)
                get_status
                ;;
            *)
                N=/etc/init.d/$NAME
                echo "Usage: $N {start|stop|restart|reload|force-reload|status}">&2
                exit 1
                ;;
        esac
        exit 0
  services:
    sysvinit:
      uwsgi:
        enabled: "true"
        ensureRunning: "true"
        files:
          - "/srv/uwsgi.ini"

 /etc/nginx/conf.d/django.conf:

    upstream django {
      server unix:///tmp/uwsgi.sock;
    }

    server {
      listen       *:443 ssl;

      ssl on;

      ssl_certificate           /srv/certs/main.crt;
      ssl_certificate_key       /srv/certs/main.key;
      ssl_session_cache         shared:SSL:50m;
      ssl_session_timeout       10m;
      ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;

      # the domain name it will serve for
      charset     utf-8;

      #Max upload size
      client_max_body_size 75M;   # adjust to taste

      # Django media
      location /media  {
            alias /srv/www/media;      # your Django project's media files
      }

        location /static {
            alias /var/www/static;     # your Django project's static files
        }

      # Finally, send all non-media requests to the Django server.
      location / {
        uwsgi_pass  django;
        uwsgi_read_timeout 7201;
        client_max_body_size 200M;
        # uwsgi_buffering off;
        include     /etc/nginx/uwsgi_params; 
        }
    }

Questions are 1. what makes this error unix:///tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream ? 2. What is the workaround to fix the error to avoid the memory consumption of Django App?

Open thoughts.

Answer 1

in nginx.conf

upstream django {
  server unix:///tmp/uwsgi.sock;
}

server {
  listen       *:443 ssl;
  ssl on;
  ssl_certificate           /srv/certs/main.crt;
  ssl_certificate_key       /srv/certs/main.key;
  ssl_session_cache         shared:SSL:50m;
  ssl_session_timeout       10m;
  ssl_prefer_server_ciphers on;

  # The domain name it will serve for
  charset     utf-8;

  #Max upload size
  client_max_body_size 75M;   # adjust to taste

  # Timeout should be equal to ngix.conf -> time keepalive_timeout 75;
  client_header_timeout 75s;

  # Django media
  location /media  {
        alias /var/www/media;      # your Django project's media files
  }

  location /static {
        alias /var/www/static;     # your Django project's static files
    }

  # Finally, send all non-media requests to the Django server.
  location / {
    uwsgi_pass  django;
   # when a client closes the connection then keep the channel to uwsgi open. Otherwise uwsgi throws an IOError
    uwsgi_ignore_client_abort on;
    uwsgi_read_timeout 120s;
    uwsgi_send_timeout 120s;
    client_max_body_size 200M;
    # uwsgi_buffering off;
    include     /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
}