Tomcat 5.5上突然出现Java堆空间错误

Question

We have a Spring 2.0.8 application in production, running on Tomcat 5.5.x and JRE 1.5.x (yeah, I know, we should upgrade, that's not the point now), with Oracle 11g as our choice of DB.

We have upgraded the application some months ago (I'd say July) and have switched from Oracle 10g to Oracle 11g in the past month or so (also changing the Oracle JDBC driver to match the database version).

We've been having serious and unexpected problems in production. As of a day ago, there have been heap space OutOfMemory errors several hours apart. This in turn either slows down response time by about a 100 times, or the users can't connect.

Our setup is:

• Windows machine to run the server
• Apache 2.2 and Tomcat 5.5 with SSO enabled, total memory: 128MB, max memory: 512MB
• Spring 2.0.8 Webapp
• Oracle 11g

Since noticing this error, this is what we tried:

• checking out the logs - there doesn't seem to be a pattern. Obviously, logs only tell you when the server is out of memory, so they show the point of not working anymore, instead of the point where the problem started
• restarting server
• reinstalling Tomcat
• increasing amount of memory Tomcat can use - this just prolonged the issue, of course Tomcat ate just as much as we gave it
• fresh installation of both the server and Apache+Tomcat
• generating heap dumps - nothing spectacular seems out of the ordinary, most memory is used for starting up the application
• checking the DB - it's fine, quick and responsive, no locks

I'm looking out for ideas on what else to do. We have this same setup in 5 different productions in total, this problematic one being with the smallest number of users and data.

Answer 1

Now that you have figured it out, I recommend that you add the following to the list of things to do REAL SOON:

• Upgrade your JVM to Java 7. Java 5 has been "end-of-lifed" which means that you won't be getting any more security patches ... unless you are on an Oracle Java support contract.

• If you can't upgrade to Java 7 ... or Java 6, then at least upgrade to the most recent patch release for Java 5 that you can get hold of.

• Upgrade to Tomcat 6 or 7, or at least to the most recent Tomcat 5.5.

• To head off problems where OutOfMemoryError causes severe slowdowns, make sure that you have the -XX:+UseGCOverheadLimit option on your JVM command line.

And, if you plan to do any significant development work on that system, consider upgrading it to Spring 3.x.

Answer 2

Ok, we figured this one out. It turns out it was a badly written SQL query which was scarcely used. Analyzing the heapdump helped find the objects that were taking up a lot of memory, and we went from there.