停止接受新的TCP连接，而不丢弃任何现有的TCP连接

Question

I have two servers listening on a TCP port behind a load balancer. The load balancer can detect if a TCP connection attempt from a client was unsuccessful and retry it to the second server without dropping that connection. I want to be able to bring any of these two servers down for maintenance without dropping a single client collection.

My servers use this code to process client requests:

ServerSocketFactory ssf = ...
ServerSocket serverSocket = ssf.createServerSocket(60000);
try {
    while (true) {
        Socket socket = serverSocket.accept();
        ...// Do the processing
    }
} catch (IOException e) {
    ...
}
...

My initial thought was to add a boolean that would be set on application shutdown and prevent new serverSocket.accept() calls while waiting for all existing connection to be processed and closed. However, new connection are being established even before the serverSocket.accept() call. Here's what I see in Wireshark if I put a breakpoint before that call. The problem is at this point as soon as I call serverSocket.close() , all such client connections get dropped. What I want to achieve is some way of telling ServerSocket to stop accept all new connections (ie only send RST for new connections or let them time out), so the load balancer can reroute them to another server, but at the same time not drop any already established connections.

Edit: I'm looking for some automated solution which wouldn't require me to change any load balancer or OS settings every time I want to update the application.

Answer 1

You could add firewall rule on the server which will block new but keep old connections active. I guess the server is Linux based? If so, you could try with:

iptables -A INPUT -p tcp --syn --destination-port <port> -j REJECT --reject-with icmp-host-prohibited

After that you can check with netstat is there any active connection and bring the application down once when there is no any:

netstat -ant|grep <port>|grep EST

After you finish with the maintenance, you can remove the firewall rule. First, list all the rules to find it:

iptables -L -n

And remove it:

iptables -D INPUT <rule number>

Answer 2

At any point when ServerSocket.accept() blocks, or ServerSocketChannel.accept() returns null, the backlog queue is empty. At that point , stop accepting and close the listening socket. Wait for all existing accepted sockets to finish their work and let the application exit at that point.

Answer 3

The easiest way to solve your problem is to put additional load balancer locally right before your application server.

Check nginx and HAproxy and chose on of them, which is better for your task. They both have a feature for graceful shutdown, which means that they stop accepting new connections but continue serving existing to the end. Another advantage is that your application doesn't require any changes in code.

Graceful shutdown for nginx :

nginx -s quit

Graceful shutdown for HAproxy :

haproxy -sf $(cat /var/run/haproxy.pid)

Answer 4

I came to the conclusion that what I'm trying to achieve is not possible on Linux. The problem is the OS completes the initial handshake with the clients by sending the SYN,ACK and ACK packet, without any control over this process by the application. After the handshake, the connection becomes established and the OS puts it in the backlog queue. As soon as the connection is established, the load balancer that I'm using (F5 BigIP) doesn't forward it to another server under any circumstances, regardless of what kind of health checks I have there. When I close the socket, already established but not yet accepted connections from the backlog queue got dropped.

However, it's possible to achieve with Windows using the SO_CONDITIONAL_ACCEPT socket option and WSAAccept function of the Windows Sockets C++ API. This option allows the application to control the initial handshake. A good explanation can be found in this answer :

When calling listen() on a port, the OS starts accepting connections on that port. This means that is starts replying SYN,ACK packets to connections, regardless if the C code has called accept() yet. ... However, on windows, the SO_CONDITIONAL_ACCEPT call allows the application to take control of the backlog queue. This means that the server will not answer anything to a SYN packet until the application does something with the connection. This means, that rejecting connections at this level can actually send RST packets to the network without creating state.

It looks like Linux doesn't have a similar feature , as described in this answer :

The three-way handshake is a part of the basic structure of tcp/ip, so it's imbeded in the stack (ie kernel level). All the non-kernel code you get your hands on operate AFTER the handshake.