Too many persistent TCP connections

Question

We have around 500 clients connected to a Linux RedHat ES 5 server. Recently it occurs, that the server still holds connections to clients which have been rebootet without stopping the application, which communicates with the server, before. A netstat on the client always returns only one established connection to the server. After a client reboot, communication runs over a new established connection. On server side sometimes the old connection is closed, sometimes it stays in state established so that we have a growing number of established connections to each client. Because various client operating systems are affected, I think that this isn't an application issue, but one of the Linux OS of the server. I tried to tune the values of

• net.ipv4.tcp_keepalive_time = 600
• net.ipv4.tcp_keepalive_intvl = 10
• net.ipv4.tcp_keepalive_probes = 9

without success. Also I tried to set the maximum file handles value from 1024 to 2048, but connections still never get closed, not even after the TCP keepalive time expires. Does somebody have an idea what could cause that strange behaviour?

Answer 1

Those settings allow you to configure the default keep-alive behavior ( when keep-alives are enabled ). However, they do not make keep-alives automatic. The feature must still be explicitly enabled on a per-socket basis via the SO_KEEPALIVE socket option.

See http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/ for details. From section 3:

Remember that keepalive support, even if configured in the kernel, is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockopt interface.