Full GC only release little memory in PSOldGen

Question

We have performed a server migration from Solaris SunOS 5.10 to Redhat Linux VM recently. JVM was upgraded from 1.5.0_22 (32-bit) to 1.6.0_06 (64-bit) However, since then, we encounter OutOfMemory error frequently. We have studied that a 64-bit JVM would require 30 - 50% more heap, so we increased our heap size from 1200MB to 2048MB and have a try. However, we still observed some OOME occured after server run for a few dates.

Upon checking the GC log, we found that Full GC happened frequently after server has started for few dates, and for each Full GC, it will only release a little memory and frequent Full GC slows down the application.

As you can see the excerpt of the GC log, almost no memory was released in PSOldGen

205023.895: [Full GC [PSYoungGen: 225919K->157256K(240960K)] [PSOldGen: 1841151K->1841151K(1841152K)] 2067071K->1998408K(2082112K) [PSPermGen: 108720K->108720K(109056K)], 6.2785770 secs] [Times: user=6.23 sys=0.01, real=6.28 secs] 
Heap after GC invocations=1638 (full 251):
 PSYoungGen      total 240960K, used 157256K [0x00002aab2e800000, 0x00002aab3e200000, 0x00002aab3e200000)
  eden space 225920K, 69% used [0x00002aab2e800000,0x00002aab38192208,0x00002aab3c4a0000)
  from space 15040K, 0% used [0x00002aab3d350000,0x00002aab3d350000,0x00002aab3e200000)
  to   space 15040K, 0% used [0x00002aab3c4a0000,0x00002aab3c4a0000,0x00002aab3d350000)
 PSOldGen        total 1841152K, used 1841151K [0x00002aaabe200000, 0x00002aab2e800000, 0x00002aab2e800000)
  object space 1841152K, 99% used [0x00002aaabe200000,0x00002aab2e7fffc8,0x00002aab2e800000)
 PSPermGen       total 109056K, used 108720K [0x00002aaaae200000, 0x00002aaab4c80000, 0x00002aaabe200000)
  object space 109056K, 99% used [0x00002aaaae200000,0x00002aaab4c2c3f8,0x00002aaab4c80000)
}  

Here is the heap usage pattern for a single OC4J instance within 24 hours, which is quite strange to me, it doesn't show a zig-zag path but instead, some random pattern.

May I know what can I do?

Server config:

Red Hat Enterprise Linux Server release 5.7 (Tikanga) 2.6.18 274.el5 (64-bit)
CPU : 8, 16GB RAM
JVM version : Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Application server : OC4J 10.1.3.5

JVM starup arguments:

//Old confing
-server -Xms1200M -Xmx1200M -XX:MaxPermSize=64M -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xnoclassgc -verbose:gc -XX:NewSize=250M -XX:MaxNewSize=250M -XX:SurvivorRatio=15 -Xconcurrentio -Xss128k

//New config
-server -Xms2048M -Xmx2048M -XX:MaxPermSize=256M -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xnoclassgc -verbose:gc -XX:NewSize=250M -XX:MaxNewSize=250M -XX:SurvivorRatio=15 -Xconcurrentio -Xss128k

Answer 1

Either this is a memory leak in the application or a bug in the Java memory management.

Since the 1.6.0_06 release there is a whole bunch of bug fixes regarding memory management and garbage collection for example these two:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6676016 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6824570

So try the following:

1. Upgrade to a newer Java 1.6.0 build. (If not possible you can change GC strategy settings from (for example) parallel to concurrent to see if that bypasses a potential Java bug.
2. Troubleshoot your application for a memory leak using a heapdump (jmap) to see what´s occupying the heap.