nginx forward proxy - failed (24: Too many open files),

Question

I'm using php curl with nginx as a proxy. here is my code:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);

echo $curl_scraped_page;

after sometime that this running the nginx load is extreamly slow and sometime it returns error 500.

the log says

failed (24: Too many open files),

some more details:

root@proxy-s2:~# ulimit -Hn
4096
root@proxy-s2:~# ulimit -Sn
1024

There is nothing else running on the server, and no other script is using this proxy.

Is it nginx bug? how to resolve it?

or

What else could it be? how can it be resolved?

• I didn't change the default nginx configuration
• Nginx restart solved the problem (temporarily I guess)

here is my nginx.conf

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    gzip  on;

    server {
        listen       8080;

        location / {
            resolver 8.8.8.8;
            proxy_pass http://$http_host$uri$is_args$args;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

top

top - 09:23:55 up 21:51,  1 user,  load average: 0.09, 0.13, 0.08
KiB Mem:    496164 total,   444328 used,    51836 free,    12300 buffers
KiB Swap:        0 total,        0 used,        0 free.   336228 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    8 root      20   0       0      0      0 S  0.0  0.0   4:57.56 rcuos/0
 4904 nobody    20   0   97796  14128   1012 R  0.0  2.8   4:19.82 nginx
    7 root      20   0       0      0      0 S  0.0  0.0   2:11.35 rcu_sched
    3 root      20   0       0      0      0 S  0.0  0.0   0:18.50 ksoftirqd/0
  832 root      20   0  139208   6808    172 S  0.0  1.4   0:13.11 nova-agent
   45 root      20   0       0      0      0 S  0.0  0.0   0:06.21 xenbus
   74 root      20   0       0      0      0 S  0.0  0.0   0:03.03 kworker/u30:1
  155 root      20   0       0      0      0 S  0.0  0.0   0:02.73 jbd2/xvda1-8
   46 root      20   0       0      0      0 R  0.0  0.0   0:02.39 kworker/0:1
   57 root      20   0       0      0      0 S  0.0  0.0   0:01.91 kswapd0
    1 root      20   0   33448   2404   1136 S  0.0  0.5   0:01.47 init
  391 root      20   0   18048   1336    996 S  0.0  0.3   0:00.97 xe-daemon
 1034 syslog    20   0  255840   2632    784 S  0.0  0.5   0:00.90 rsyslogd
 1107 root      20   0   61364   3048   2364 S  0.0  0.6   0:00.73 sshd
   40 root      rt   0       0      0      0 S  0.0  0.0   0:00.29 watchdog/0
  316 root      20   0   19472    456    252 S  0.0  0.1   0:00.12 upstart-udev-br
    6 root      20   0       0      0      0 S  0.0  0.0   0:00.11 kworker/u30:0
 1098 root      20   0   23652   1036    784 S  0.0  0.2   0:00.08 cron
 7935 root      20   0  105632   4272   3284 S  0.0  0.9   0:00.07 sshd
  330 root      20   0   51328   1348    696 S  0.0  0.3   0:00.06 systemd-udevd
 7953 root      20   0   22548   3428   1680 S  0.0  0.7   0:00.05 bash
  678 root      20   0   15256    524    268 S  0.0  0.1   0:00.04 upstart-socket-
 8647 root      20   0   25064   1532   1076 R  0.0  0.3   0:00.03 top

mpstat

root@proxy-s2:~# mpstat
Linux 3.13.0-55-generic (proxy-s2)      07/09/2015      _x86_64_        (1 CPU)

09:22:17 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:22:17 AM  all    0.94    0.00    1.63    0.16    0.00    2.16    0.92    0.00    0.00   94.20

iostat

root@proxy-s2:~# iostat
Linux 3.13.0-55-generic (proxy-s2)      07/09/2015      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.94    0.00    3.80    0.16    0.92   94.19

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
xvdc              0.01         0.02         0.00       1710          0
xvda              3.16         4.19        88.56     322833    6815612

Answer 1

Please try below ,do the following changes in your limits.conf.

vi /etc/security/limits.conf

For open files

• soft nofile 64000

• hard nofile 64000

  For max user processes

• soft nproc 47758

• hard nproc 47758

  For max memory size

• soft rss unlimited

• hard rss unlimited

  For virtual memory

• soft as unlimited

• hard as unlimited

Answer 2

Just put this on atop of Nginx configuration file:

worker_rlimit_nofile 40000;

events {
    worker_connections  4096;
}

Answer 3

I think I found the problem:

here is the nginx error.log

2015/07/09 14:17:27 [error] 15390#0: *7549 connect() failed (111: Connection refused) while connecting to upstream, client: 23.239.194.233, server: , request: "GET http://www.lgqfz.com/ HTTP/1.1", upstream: "http://127.0.0.3:80/", host: "www.lgqfz.com", referrer: "http://www.baidu.com"
2015/07/09 14:17:29 [error] 15390#0: *8121 connect() failed (111: Connection refused) while connecting to upstream, client: 204.44.65.119, server: , request: "GET http://www.lgqfz.com/ HTTP/1.1", upstream: "http://127.0.0.3:80/", host: "www.lgqfz.com", referrer: "http://www.baidu.com"
2015/07/09 14:17:32 [error] 15390#0: *8650 connect() failed (101: Network is unreachable) while connecting to upstream, client: 78.47.53.98, server: , request: "GET http://188.8.253.161/ HTTP/1.1", upstream: "http://188.8.253.161:80/", host: "188.8.253.161", referrer: "http://188.8.253.161/"

It was a DDOS attack on my PROXY that I stopped by allowing only my IP to access the PROXY.

I found it to be common lately - when u crawl a site, and the site identify you as a crawler, it will sometime DDOS attack your proxy until they go black. One example of such site is amazon.com