Increasing class loaded count in Spring Boot application deployed as WAR on Tomcat

Question

We have a Spring Boot application running on Tomcat, it is a RESTful web service. The same WAR file is deployed on 3 Tomcat instances in our test environment as well as Production environment. While running performance test we noticed a peculiar problem with some servers. Some of the servers stop responding after processing about 2500 requests. The issue happens on 2 out of 3 Production servers and happens on 1 out of 3 Test servers.

On the servers that have the issue, we noticed on our JVM monitoring that the classes loaded count keeps increasing whenever we are running the performance test. The class loaded count goes from 20k to around 2 million. When the class count reaches close to 2million the JVM monitoring also shows that the GC is taking too long, more than 40 seconds. Once it reaches that point, the application would stop responding. The applications throws an OutOfMemoryException “Compressed class space”. If we continue sending more requests, we can see from the application logs that the service is still receiving requests but stops processing midway.

On the other servers without the issue, the class loaded count stays at a constant 20k. And the GC is normal too, taking less than 1 seconds.

Others testing and behaviors we have noticed -

1. The issue happens on local Tomcat instances installed on Windows PC. The servers are on Linux. The issue happens on both OpenJDK and Oracle JDK 1.8.
2. We verified the Tomcat instances are equal to each other - we even cloned from the working servers and put them on the bad servers.
3. Tested with different GC policies - PS, CMS and G1, and the issues happens on all three.
4. Tested by running the application as a standalone Spring Boot JAR and the issue goes away. The class count stays constant and GC behaves normally.
5. The application is currently using JAXB libraries to perform XML marshalling/unmarshalling and we found places in the code where we can optimize the code. Refactoring the code and moving to Jackson library is another option.

My questions are -

• What would be causing the difference between multiple servers when we are deploying the same WAR file?.
• What would be causing the difference between the application running as WAR deployed on Tomcat versus running as standalone Spring boot application?
• If we take a heap dump of the JVM or do a profiling, what are the things to look out for?

Answer 1

So it turns out this was due to jaxb 2.1 jar in our classpath. Thanks to Mark for pointing out the known bug with jaxb.

Our application did not explicitly have the jaxb-impl as a dependency, so it was hard to see at first. Upon looking at the Maven dependency tree, we found out two different versions were being loaded from other project and libraries. Our application had jaxb-impl versions 2.1 and 2.2.6 in the classpath. We put the 2.1 version as an exclusion in our application's pom.xml and that fixed the issue.

My guess is that different servers were loading different versions upon the application startup. That could be why some servers were working fine and others that loaded the 2.1 version had issues. Similarly with running as a standalone Spring boot app, it might have loaded the 2.1 version.