TIME_WAIT and EADDRINUSE on Linux

Question

I have a system in which a server process, which runs in the background, is controlled by a control program.

The control program is a simple script which performs one command and then exits. For the Run command it creates a new server process; for the others (including Shutdown) it sends the command to the server via a reserved control port.

The control program creates a socket, connects to the server's control port, sends one character of data (a command), reads the reply from the socket, closes the socket, displays the reply to the user and exits. The server accepts the connection on the control socket, reads a character of data, sends a reply, closes the child socket and continues listening. When the server shuts down it also closes the parent control socket.

This has been working on Windows XP / Python 2.6.6 for a number of years. Recently we have tried porting to Linux (Ubuntu 16.04.2 GNU/Linux 4.4.0-62-generic x86_64 / Python 2.7.12), but there the Restart command (Shutdown immediately followed by Run) fails: when the new server process tries to bind the control socket to the control port, it gets EADDRINUSE.

The output from netstat shows that the connection used for the Shutdown command remains in TIME_WAIT state, and continues so for about two minutes.

I have reviewed previous posts on this topic. I have tried setting SO_REUSEADDR and/or SO_REUSEPORT on the server's control socket. I have tried enhancing the protocol between server and control program to ensure that the control program closes its side of the connection first, but so far I have not found a combination that works. I wonder whether there is any solution.

Given that both server and control program run on the same machine, details of both sides of the connection will be in the operating system's state table, and one or other must be left in TIME_WAIT.

Does an entry for the control program prevent the server from binding to the port?

I note that on Windows also a connection is left in TIME_WAIT state, but on Windows that does not prevent the new server process from binding to the same control port.

Answer 1

I have answered my own question. To get the desired behaviour, I need to set SO_REUSEADDR (only) on the server's control socket (only). It is not necessary to use SO_REUSEPORT. It is not necessary to do anything to the control program. It makes no difference which side closes the connection first.

My first attempt did not work because I made a stupid mistake. (I used the wrong variable.)