為什么nginx不接受來自apache bench的超過1024個連接

Question

我有一個clojure / jetty服務器（在端口8081上）並使用nginx（端口8080）代理它。 我一直試圖在其代理角色中單獨對clojure應用程序和nginx進行基准測試。

當我直接對clojure運行測試時，我可能會Connection reset by peer ，10次運行。 通常，測試完成，性能可以接受。

$ ulimit -n 4096
$ ab -n 20000 -c 2048 -k localhost:8081
...
Concurrency Level:      2048
Time taken for tests:   8.713 seconds
Complete requests:      20000
Failed requests:        0
Keep-Alive requests:    20000
Total transferred:      15160000 bytes
HTML transferred:       11720000 bytes
Requests per second:    2295.43 [#/sec] (mean)
Time per request:       892.208 [ms] (mean)
Time per request:       0.436 [ms] (mean, across all concurrent requests)
Transfer rate:          1699.16 [Kbytes/sec] received
...

我開始測試完整的本地配置，端口8080上的nginx和8081上的clojure。事情進展順利，直到我超過1024個並發連接。

我注意到，使用ss -tl ，接收隊列沒有尖峰，或者至少如果它們是閃存的話。 但我確實發現，使用netstat -s ，正在發送大量TCP RST。 有時， dmesg告訴我，看起來有一個SYN泛濫。 此外，nginx響應HTTP狀態499，這應該表明客戶端關閉連接...

所以，我的診斷是交叉的，apache bench和nginx聲稱對方關閉了連接！？

$ ulimit -n 4096
$ ab -n 20000 -c 2048 -k localhost:8080
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
apr_socket_recv: Connection reset by peer (104)
Total of 26 requests completed

組態

我為nginx和clojure設置ulimit max open files為4096。

無用的網絡變化

net.core.netdev_max_backlog=30000
# yes, we are using jumbo frames
net.ipv4.tcp_mtu_probing=1

net.core.somaxconn=4096
net.ipv4.ip_local_port_range=4096     61000
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_max_syn_backlog=2048

/etc/nginx/nginx.conf

user www-data;
worker_processes  2;
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log;
pid        /run/nginx.pid;

events {
  worker_connections  2048;
  use epoll;
}

http {

  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;


  access_log    /var/log/nginx/access.log;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_requests 1000;
  keepalive_timeout  65;


  gzip  on;
  gzip_http_version 1.0;
  gzip_comp_level 2;
  gzip_proxied any;
  gzip_vary off;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/rss+xml application/atom+xml text/javascript application/javascript application/json text/mathml;
  gzip_min_length  1000;
  gzip_disable     "MSIE [1-6]\.";


  variables_hash_max_size 1024;
  variables_hash_bucket_size 64;
  server_names_hash_bucket_size 128;
  types_hash_max_size 2048;
  types_hash_bucket_size 64;



  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

在/ etc / nginx的/啟用的站點 - / UPS

upstream ups {
  server localhost:8081 fail_timeout=0;
}


server {
  listen 8080 backlog=1024;

  server_name example.com;

  proxy_buffer_size   128k;
  proxy_buffers   4 256k;
  proxy_busy_buffers_size   256k;

  client_max_body_size 3M;

  large_client_header_buffers 4 128k;

  proxy_read_timeout 300;
  proxy_send_timeout 300;
  send_timeout 300;
  keepalive_timeout 300;

  server_tokens off;

  access_log /var/log/nginx/ups_access.log enhanced-combined;
  error_log /var/log/nginx/ups_error.log;
  root /apps/ups/current/public/;

  error_page 403 /errors/403_maintenance.html;
  error_page 500 /errors/500.html;
  location ^~ /errors/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Request-Id $request_uuid;
    satisfy any;
    allow all;
  }

  location / {

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Request-Id $request_uuid;

    if ($http_x_forwarded_proto = 'http') {
      rewrite ^ https://$host$request_uri? permanent;
    }


    if (-f $request_filename/index.html) {
      rewrite (.*) $1/index.html break;
    }

    if (-f $request_filename.html) {
      rewrite (.*) $1.html break;
    }

    if (!-f $request_filename) {
      proxy_pass http://ups;
      break;
    }
  }
}

Answer 1

@Terra幾乎得到了解決問題的答案。

• accept_mutex off;
• worker_connections 4096;

我嘗試自己添加每個更改並重新加載，但我仍然看到了同樣的錯誤。 直到我改變了兩個，我才能超過1024個連接。

看起來，由於代理，我需要有兩倍的工作連接，因為我打算接受。