NodeJS stopping to accept new connections after 1000ish concurrent connections

Question

My application has started to receive more and more traffic now some days I am hitting around 1000 concurrent connections. At that point for some reason my NodeJS server stops to accept my connections, they are just closed.

The NodeJS server is behind a HAProxy server so I can see on the stats page of HAProxy that when I am right around 1000 concurrent connections I am getting Resp errors the exact error is:

"Connection resets during transfers"

This leads me to believe that NodeJS is simply refusing new connections. At that point I started to research how to up the connections that node can handle so I did the following:

Added this to my sysctl.cfg

fs.file-max = 2097152
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_nonlocal_bind = 1

The way I start my Node server is like:

Login to the EC2 from my terminal on Mac OSX, and write:
sudo su
cd node/
ulimit -n 10000
forever start server.js

Oh and note that on the HAProxy server I got the same entires in the sysctl.cfg and also have upped the maxconn to 128000

Im pretty sure the problem here is the node server, but cannot understand why node still seems to max out at 1000ish connections?

Any ideas here? Im out of ideas of what to try.

Answer 1

Found the solution.

I had to install the npm module posix

In my app i added

var posix = require('posix');

posix.setrlimit('nofile', { soft: 15000 });

I did a simple load test where i connected 15k tcp connections and after 15k it gave me the same error as above.

Hope this can help someone else having this issue.