Outgoing client TCP ports get blocked when there are too many connections

Question

I have a distribution system for doing file operation with and running shell commands on target client machines on windows. and I use a custom TCP endpoint for connecting to the windows service which is resident on the server.

Now I've created this tool to create numerous instances of that agent(client) on one machine and run a certain job-set against all of them from the server. The problem is that all outgoing TCP ports on the client machine gets blocked after launching more than a few hundred agents. each agent is using a dynamic port and is listening to a single server port.

Say, i got 2000 agents running on ports 2000-3999 and all are listening to port 5111 on the server. The error message i'm receiving in windows event log goes like this:

TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

When this occurs this machine cannot use any TCP port anymore. I did try changing some of the TCP default behavior in registry but to no avail. The interval between opening connections is between 1 to 5 seconds.

Any workaround for managing the optimal delay and/or somehow make windows trust the application irrespective of the aggressive network activity required for the test?

Answer 1

Turns out if you open connections without a proper gap in between, all ports on the client will get blocked due to aggressive behavior. Finally i got to connect all my agents by increasing the delay between each connection from 1000 ms to 3000 ms. I am still to figure out the dynamics of this though, and probably opening agents in parallel threads rather than processes would be a better solution after all. High number of processes with the same name seemingly doesn't appeal to the OS.