permgen garbage collection takes multiple Full GC

Question

We are running grails and we are noticing multiple full garbage collections are needed to clear out the permgen space.

2013-06-06T16:11:27.016+0000: 32582.145: [Full GC 32582.145: [CMS2013-06-06T16:11:45.404+0000:     32600.532: [CMS-concurrent-mark: 21.403/86.063 secs] [Times: user=48.44 sys=0.63, real=86.07 secs]
(concurrent mode failure): 7585874K->7290466K(10145024K), 57.9230770 secs] 7866094K->7290466K(10451712K), [CMS Perm : 261766K->261702K(262144K)] icms_dc=30 , 57.9232150 secs] [Times: user=57.97 sys=0.00, real=57.93 secs]
2013-06-06T16:12:25.183+0000: 32640.311: [GC [1 CMS-initial-mark: 7290466K(10145024K)] 7385976K(10451712K), 0.0880890 secs] [Times: user=0.09 sys=0.00, real=0.08 secs]
2013-06-06T16:12:25.271+0000: 32640.400: [CMS-concurrent-mark-start]
2013-06-06T16:12:25.427+0000: 32640.555: [GC 32640.556: [ParNew: 272640K->10006K(306688K), 0.0622620 secs] 7563106K->7300472K(10451712K) icms_dc=30 , 0.0624140 secs] [Times: user=0.24 sys=0.00, real=0.06 secs]
2013-06-06T16:12:25.734+0000: 32640.863: [GC 32640.863: [ParNew: 282646K->13476K(306688K), 0.0648770 secs] 7573112K->7303942K(10451712K) icms_dc=30 , 0.0650170 secs] [Times: user=0.26 sys=0.00, real=0.07 secs]
2013-06-06T16:12:26.013+0000: 32641.142: [GC 32641.142: [ParNew: 286116K->11277K(306688K), 0.0607460 secs] 7576582K->7301743K(10451712K) icms_dc=30 , 0.0608870 secs] [Times: user=0.25 sys=0.00, real=0.06 secs]
2013-06-06T16:12:32.449+0000: 32647.577: [GC 32647.577: [ParNew: 283917K->17560K(306688K), 0.0672260 secs] 7574383K->7308026K(10451712K) icms_dc=30 , 0.0673710 secs] [Times: user=0.27 sys=0.00, real=0.07 secs]
2013-06-06T16:12:33.107+0000: 32648.236: [GC 32648.236: [ParNew: 290200K->22523K(306688K), 0.0686820 secs] 7580666K->7312989K(10451712K) icms_dc=30 , 0.0688200 secs] [Times: user=0.28 sys=0.00, real=0.07 secs]
2013-06-06T16:12:33.845+0000: 32648.974: [Full GC 32648.974: [CMS2013-06-06T16:12:52.516+0000: 32667.645: [CMS-concurrent-mark: 21.346/27.245 secs] [Times: user=27.55 sys=0.14, real=27.25 secs]
(concurrent mode failure): 7290466K->7293289K(10145024K), 57.7092090 secs] 7523492K->7293289K(10451712K), [CMS Perm : 262143K->262143K(262144K)] icms_dc=30 , 57.7093560 secs] [Times: user=57.76 sys=0.00, real=57.71 secs]
2013-06-06T16:13:31.557+0000: 32706.686: [Full GC 32706.686: [CMS: 7293289K->6960613K(10145024K), 37.1325250 secs] 7293289K->6960613K(10451712K), [CMS Perm : 262143K->91949K(262144K)] icms_dc=30 , 37.1326670 secs] [Times: user=37.19 sys=0.00, real=37.14 secs]

The first collects only 64K, the second collects nothing and then finally, the third is able to collect 170194K

JAVA_OPTIONS:
-XX:+CMSClassUnloadingEnabled 
-XX:+CMSPermGenSweepingEnabled 
-XX:+UseConcMarkSweepGC
-XX:MaxPermSize=256m 
-XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps 
-XX:+PrintGCDateStamps 
-verbose:gc,sizes 
-XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=80 
-XX:+DisableExplicitGC 
-XX:+CMSIncrementalMode 
-XX:+UseParNewGC  
-Xms10g -Xmx10g

Also, is there anyway to tell get the garbage collector to do an incremental sweep of the permgen space? We only see permgen being reduced during full collections.

Answer 1

Let me do a clarification about Concurrent Mark Sweep and it's incremental mode algorithm. CMS incremental mode was introduced to avoid CPU starvation on single core servers while background GC is running. Usage of incremental CMS is discouraged.

Incrementally mode does not free memory incrementally, it just doing long sleeps during mark phase of mark-sweep algorithm.

-XX:+CMSPermGenSweepingEnabled is deprecated and synonymous to -XX:+CMSClassUnloadingEnabled

Incremental mode somewhat hiders dead object detection and my be a problem.

In addition, if any of classes (to be unloaded) has finilazer this also could explain 2 collections (JVM cannot unload individual classes, whole classloader is unloaded, so any of its classes can prevent collection).

I would advise you to properly size heap and perm gen, and configure CMS to be more aggressive if you want to keep that level of heap utilization. In my blog you can find few advises for tuning CMS.

If however, you run time is actively producing and abandoning class loaders tuning GC may not be enough.

I was facing similar issues with automated tests (each test were loading same classes in dedicated class loader for sake of isolation). Test were generally unstable, throwing OOME in perm gen. Solution was to force JVM to clean perm gen by polluting it with garbage data ( here is snippet of code ). It was causing Full GC though - probably not something you would like to see.

BTW There is also -XX:CMSClassUnloadingMaxInterval=N flag which let JVM collect Perm gen only every Nth collection. But this is not your case, cause perm gen is full.

Answer 2

One way to identify what is being collected in that last FullGC collection is to print class histograms before/after Full GC: -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC .

This way you can compare histograms from all collections and identify which classes are collected in the last one.

To your second question considering PermGen, the usual advice is to identify the sufficient size of the PermGen for your application/workload and stick with that. You need to investigate why there is so many objects being placed into the PermGen.