tomcat cpu usage goes 100% with virtual ip configuration

Question

We have our application running on tomcat server. It works smoothly but when load balance server is configured with this application, cpu usage goes to 100%.

Strange things is, when we look at the thread dump, there is no suspicious thread activity. Most of the threads are waiting or parked even when cpu usage of java process is 100%. It simply goes to 2% to 5% when we remove virtual ip configuration.

I learned from load balancer/Virtual ip configuration that it probe the server port which is 8080 at every 5 seconds just to check if the port is alive or not. But I assume that this is just port scanning nothing else. There is no mark or such activity in thread dump neither in catalina.log during this process.

Does any one have faced this kind of situation? Kindly provide your thoughts as I am pointless for further debugging this problem.

Let me know if more information is required. I will update the question with the details.

Thanks,

Gaurav

Answer 1

Please take a look at this post too: https://serverfault.com/a/830052

Poster Inna found out that a single call to https://server.tld:8080 could get the Tomcat server to use 100%. I can confirm this, because I just reproduced this on one of our servers (on staging platform luckily). We're running Ubuntu 12.02 with Tomcat 6. They just released an update yesterday which fixes a regression. Not sure if this solves the problem.

The proposed solution is to revert to a "known-good" version of Tomcat, but the issue might be solved in last nights update/patch too. I will update this post when I have more info.

[Update 1]
Most recent updates known on 2017-02-03 11:30 CET did not fix the problem. I've updated our staging server with all updates and it still has the same issue.