Packets dropped by kernel during load testing of haproxy -> node.js -> db

Question

we are in the process of load testing our game server stack, which consists of haproxy/ssl -> node.js (app servers) -> database

The calls are gets which then node.js passes to the db then returns data back to the client, how ever during various testing we are seeing various odd issues that we cant seem to track down.

While running tcpdump 'tcp[13] & 4!=0' | grep http-alt

We see various things in the log:

23:35:16.049886 IP xx > xx.http-alt: Flags [R], seq 2946709660, win 0, length 0 23:35:16.668370 IP xx.62737 > xx.http-alt: Flags [R.], seq 4204676699, ack 3037703652, win 68, options [nop,nop,TS val 153103530 ecr 133109021], length 0 23:36:06.030500 IP xx.40290 > xx.http-alt: Flags [R.], seq 431162404, ack 2610577956, win 68, options [nop,nop,TS val 153115871 ecr 133121361], length 0

Exiting the command we see

1662286 packets received by filter 9 packets dropped by kernel

our load testing apps also returns errors back to us.. along the lines of this

T000023 --- loop started --- 19 Aug 2013 18:09:07.975 ---r" T000023r" T000023 <<< user_id = nullr" T000023 <<< ext_userid = nullr" T000023 <<< Public_token_put = nullr" T000023 <<< Private_token_prt = nullr" T000023 >>> execute plug-in #1: GenerateRandomCharactersr" T000023r" T000023r" T000023 # Page #1: Registrationr" T000023 # ---------------------r" T000023r" T000023 [1] GET https://someurl.com:2222/?method=somemethod&type=reg=somerandomid ...r" T000023 -8 (Network Connection aborted by Server), [No Content Type], --- bytes, --- ms, * Failed at 'Open Network Connection', java.net.SocketException: Connection resetr" T000023 error: expected HTTP status: 200 <> received: -8 (Network Connection aborted by Server), [No Content Type], --- bytes, --- ms, * * Failed at 'Open Network Connection', java.net.SocketException: Connection resetr

Any thoughts on where to start..

servers are ubuntu 12.04 server with one tweak to sysctl

net.netfilter.nf_conntrack_max = 262144

Answer 1

The tcpdump errors are a red haring (meaning they are not related to your problem). TCPDump tries to stay out of the way of impacting system performance. If the system would get backed up for something that tcpdump is doing then it will simply drop the packet instead of inspecting it. This does not mean that the system or network stack is dropping packets. Try adding the "-n" switch to tcpdump.

-n     Don't convert addresses (i.e.,  host  addresses,  port  numbers, etc.) to names.

Unless you have a caching DNS server like dnsmasq installed, nscd is more than likely disabled. This means that for every connection that comes in tcpdump will try to do a reverse dns lookup. I bet that by adding the -n your "dropped packets" go away.

Your real problem sounds to me like you are either overwhelming system resources (cpu, memory, etc) or you are exceeding your ephemeral ports/max connections in haproxy. Also it is very well documented that conntrack + haproxy is a recipe for terrible performance.

1. Disable conntrack
2. Check your max connections config in haproxy
3. Look at the haproxy stats page (if it is not configured, configure it - it is awesome!)
4. Enable haproxy logging. I run a very large haproxy implementation using version 1.5.19. We have been using 1.5.x for almost a year now with no issues. One thing that I have learned is that haproxy logging is always 100% reliable. Sometimes I misinterpreted the log entry, but it was always accurate. In scenarios like this where you are trying to track down a connection problem, the haproxy logging will simply tell you why it is failing.
5. Check the max open file handles for the haproxy process. To do this

cat /proc/(haproxy PID)/limits

Good luck.

PS. Enable stats page, it is awesome! Also, enable haproxy logging, it is also awesome! Between these two you will be able to identify the connection issue very quickly.