Tomcat behavior for CLOSE_WAIT connections

Question

The acceptCount queue is maintained at OS level. Lets say we have a acceptCount queue of 10K.

Situation - The server is unable to process the request or is taking a long time due to one of the dependencies or network issue and during the time all the clients timeout. Eventually the queue has 10K CLOSE_WAIT connections. Now if the service is backup again and start processing. Would it clean up the CLOSE_WAIT connection queue? How will tomcat behave in such situation.

Answer 1

a few tips about CLOSE_WAITS and reaching max connections/thread count, (I had similar problems in WebLogic server, but these are general tips)

to release stuck threads due to network issues: (if allowed according to the business)

1. set timeout value to all remote calls (eg DB/ HTTP/ EJB/..)
2. check for Terminating/Killing long running threads (requests)

better check closely and get to the root-cause, before trying any workarounds.

CLOSE_WAITS should clear after heavy-load period is over in case this is the issue, but in case of stuck threads you might need a restart ASAP to solve the issue.

Answer 2

The acceptCount queue is maintained at OS level.

By this I assume you mean the listen backlog queue.

Let's say we have an acceptCount queue of 10k.

There is no way to tell. You can hint to the operating system how long you would like it to be, but the operating system can adjust it both up and down, and there is no API to tell you what the actual length is.

Eventually the queue has 10k CLOSE_WAIT connections.

No it doesn't. CLOSE_WAIT states are not queued anywhere, and the listen backlog queue has nothing to do with CLOSE_WAIT whatsoever.

Would it clean up the CLOSE_WAIT queue?

There is no such queue to clean up.

If what you are really asking is would restarting the process clear all the ports in CLOSE_WAIT state, the answer is that exiting the process which held them would do that.

However so does letting Tomcat detect all the closed connections and close them itself. If you have lots of these, the real issue is why it isn't doing that automatically. It should.

If this is caused by one request being forwarded somewhere, it sounds like something is synchronized internally that shouldn't be synchronized, in your application.