奇怪的Java JVM内存不足

Question

I have an application running under jboss5. It has a user login side as well as background quartz jobs. One background job makes a soap call and pull down a large object to parse. It uses a good amount of memory.

I see a pattern of when I get an OOM exception like this:

2013-06-04 21:44:36,855 ERROR [STDERR] (QuartzScheduler_Scheduler-NON_CLUSTERED_MisfireHandler) java.lang.OutOfMemoryError: Java heap space
2013-06-04 21:44:36,855 ERROR [STDERR] (http-0.0.0.0-80-9) Exception in thread "http-0.0.0.0-80-9" 
2013-06-04 21:44:36,855 ERROR [STDERR] (http-0.0.0.0-80-9) java.lang.OutOfMemoryError: Java heap space
2013-06-04 21:44:36,855 ERROR [STDERR] (Session Monitor) Exception in thread "Session Monitor" 
2013-06-04 21:44:36,855 ERROR [STDERR] (Monitor Runner) java.lang.OutOfMemoryError: Java heap space
2013-06-04 21:44:36,855 ERROR [STDERR] (Monitor Runner)     at java.util.Arrays.copyOf(Arrays.java:2219)
2013-06-04 21:44:36,855 ERROR [STDERR] (Monitor Runner)     at java.util.ArrayList.toArray(ArrayList.java:329)
2013-06-04 21:44:36,855 ERROR [STDERR] (Monitor Runner)     at java.util.ArrayList.<init>(ArrayList.java:151)
2013-06-04 21:44:36,855 ERROR [STDERR] (Monitor Runner)     at com.icesoft.util.MonitorRunner$1.run(MonitorRunner.java:54)

This job runs nightly. When nobody is using the UI for days, I'll get the OOM after a few days. But when people use the UI application daily, I can go for over a month or more and never see the OOM problem.

It seems like there is something good that happens when the application is used, but I have no idea what. Does anyone have an idea of where to start looking and what to try?

We're using jdk 1.7.0.11 and javaopts are

set "JAVA_OPTS=-Xrs -Xms256M -Xmx4096M -XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC "

Thank you,

Jim

Answer 1

Use -XX:+HeapDumpOnOutOfMemoryError JVM argument then open the Heap Dump in MAT. MAT will show you the suspects of the OOM, you can even walk through the heap yourself and identify which objects occupy the most of the heap. This way you can easily find the culprit of the problem.

Also, it would help enabling GC logging so you can see if there are any patterns that lead to the OOM.