What is the reason behind full GC?

Question

Every half an hour Full GC is happening. The first suspect was -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 but -XX:+DisableExplicitGC is already configured.

Java version:

java version "1.6.0_24" OpenJDK Runtime Environment (IcedTea6 1.11.8) (rhel-1.56.1.11.8.el6_3-x86_64) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

What is the reason behind this abnormal behavior?

-D[Standalone] -server 
-XX:+UseCompressedOops 
-Dsun.rmi.dgc.client.gcInterval=1800000 
-Dsun.rmi.dgc.server.gcInterval=1800000 
-Djava.net.preferIPv4Stack=true 
-Djboss.modules.system.pkgs=org.jboss.byteman
-Djava.awt.headless=true
-XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/log 
-Xloggc:/log/gc_23-Dec-2015_23.32.10CST.log 
-XX:+PrintGCDetails 
-XX:+PrintGCDateStamps
-Djava.security.egd=file:/dev/urandom 
-XX:+DoEscapeAnalysis
-XX:+CMSParallelRemarkEnabled 
-XX:+UseBiasedLocking 
-XX:+UseLargePages 
-XX:+CMSConcurrentMTEnabled 
-XX:SurvivorRatio=8 
-XX:TargetSurvivorRatio=90 
-XX:MaxTenuringThreshold=15 
-XX:NewRatio=3 
-XX:PermSize=1024m 
-Dorg.jboss.resolver.warning=true 
-XX:ReservedCodeCacheSize=128m 
-XX:+UseCodeCacheFlushing 
-XX:+UseConcMarkSweepGC 
-XX:+UseParNewGC 
-XX:+DisableExplicitGC 
-Dnet.sf.ehcache.skipUpdateCheck=true 
-Xss1024k -Xms8192m 
-Xmx8192m 
-XX:MaxPermSize=1024m 
-Dremoting.bind_by_host=false 
-Dclient.encoding.override=UTF-8 
-Dfile.encoding=UTF-8 
-Dorg.jboss.as.logging.per-deployment=false 
-Dorg.jboss.boot.log.file=/log/boot.log 
-Dlogging.configuration=file:/configuration/logging.properties 
-jar /jboss/jboss-eap-6.0/jboss-modules.jar 
-mp /jboss/jboss-eap-6.0/modules:/modules 
-jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone 
-Djboss.home.dir=/jboss/jboss-eap-6.0 
-Djboss.server.base.dir=
-Dorg.apache.el.parser.COERCE_TO_ZERO=false

and the log is

2015-12-23T09:22:54.660-0600: 21934.898: [GC 21934.898: [ParNew: 1887488K->190826K(1887488K), 0.0698680 secs] 3712766K->2016104K(8178944K), 0.0703930 secs] [Times: user=0.76 sys=0.00, real=0.07 secs] 
2015-12-23T09:23:35.235-0600: 21975.473: [Full GC 21975.473: [CMS: 1825278K->1524650K(6291456K), 6.9749440 secs] 2586494K->1524650K(8178944K), [CMS Perm : 500222K->500167K(1048576K)], 6.9753390 secs] [Times: user=6.97 sys=0.01, real=6.98 secs] 
2015-12-23T09:25:35.427-0600: 22095.664: [GC 22095.665: [ParNew: 1677821K->141297K(1887488K), 0.0310160 secs] 3202471K->1665947K(8178944K), 0.0315740 secs] [Times: user=0.27 sys=0.00, real=0.03 secs] 
2015-12-23T09:25:58.707-0600: 22118.944: [Full GC 22118.945: [CMS: 1524650K->1520964K(6291456K), 6.2934290 secs] 2023419K->1520964K(8178944K), [CMS Perm : 500192K->500167K(1048576K)], 6.2938200 secs] [Times: user=6.29 sys=0.00, real=6.30 secs] 
2015-12-23T09:28:02.317-0600: 22242.555: [GC 22242.555: [ParNew: 1677824K->48477K(1887488K), 0.0338780 secs] 3198788K->1569442K(8178944K), 0.0343380 secs] [Times: user=0.24 sys=0.00, real=0.04 secs] 
2015-12-23T09:28:21.967-0600: 22262.205: [Full GC 22262.205: [CMS: 1520964K->1522394K(6291456K), 6.4743040 secs] 1920487K->1522394K(8178944K), [CMS Perm : 500200K->500167K(1048576K)], 6.4747140 secs] [Times: user=6.46 sys=0.01, real=6.48 secs] 
2015-12-23T09:30:45.561-0600: 22405.799: [GC 22405.800: [ParNew: 1677824K->44982K(1887488K), 0.0303550 secs] 3200218K->1567376K(8178944K), 0.0308010 secs] [Times: user=0.23 sys=0.00, real=0.03 secs] 
2015-12-23T09:30:46.792-0600: 22407.030: [Full GC 22407.030: [CMS: 1522394K->1519090K(6291456K), 6.0919040 secs] 1586823K->1519090K(8178944K), [CMS Perm : 500205K->500169K(1048576K)], 6.0923080 secs] [Times: user=6.08 sys=0.00, real=6.10 secs] 
2015-12-23T09:32:50.752-0600: 22530.990: [GC 22530.990: [ParNew: 1677824K->37316K(1887488K), 0.0293470 secs] 3196914K->1556406K(8178944K), 0.0298840 secs] [Times: user=0.25 sys=0.00, real=0.03 secs] 
2015-12-23T09:33:11.243-0600: 22551.480: [Full GC 22551.481: [CMS: 1519090K->1518714K(6291456K), 6.9639780 secs] 1843400K->1518714K(8178944K), [CMS Perm : 500215K->500153K(1048576K)], 6.9644400 secs] [Times: user=6.96 sys=0.00, real=6.97 secs] 
2015-12-23T09:34:38.447-0600: 22638.685: [GC 22638.685: [ParNew: 1677824K->47923K(1887488K), 0.0236300 secs] 3196538K->1566637K(8178944K), 0.0240750 secs] [Times: user=0.20 sys=0.00, real=0.03 secs] 
2015-12-23T09:35:36.788-0600: 22697.026: [Full GC 22697.026: [CMS: 1518714K->1519053K(6291456K), 7.3073640 secs] 2398945K->1519053K(8178944K), [CMS Perm : 500186K->500133K(1048576K)], 7.3077820 secs] [Times: user=7.30 sys=0.00, real=7.30 secs] 
2015-12-23T09:37:15.722-0600: 22795.960: [GC 22795.961: [ParNew: 1677824K->49053K(1887488K), 0.0368360 secs] 3196877K->1568107K(8178944K), 0.0374250 secs] [Times: user=0.29 sys=0.01, real=0.03 secs]
2015-12-23T09:39:06.235-0600: 22906.473: [GC 22906.473: [ParNew: 1726877K->75704K(1887488K), 0.0348560 secs] 3245931K->1594758K(8178944K), 0.0353970 secs] [Times: user=0.33 sys=0.00, real=0.04 secs] 
2015-12-23T09:40:51.942-0600: 23012.180: [GC 23012.180: [ParNew: 1753528K->118268K(1887488K), 0.0509830 secs] 3272582K->1637322K(8178944K), 0.0515380 secs] [Times: user=0.50 sys=0.00, real=0.05 secs] 
2015-12-23T09:42:47.051-0600: 23127.289: [GC 23127.289: [ParNew: 1796092K->164537K(1887488K), 0.0524430 secs] 3315146K->1683590K(8178944K), 0.0530040 secs] [Times: user=0.55 sys=0.00, real=0.06 secs] 
2015-12-23T09:45:08.101-0600: 23268.338: [GC 23268.339: [ParNew: 1842361K->167363K(1887488K), 0.0589320 secs] 3361414K->1686416K(8178944K), 0.0594480 secs] [Times: user=0.64 sys=0.01, real=0.06 secs] 
2015-12-23T09:47:10.946-0600: 23391.184: [GC 23391.184: [ParNew: 1845187K->209664K(1887488K), 0.0771020 secs] 3364240K->1733317K(8178944K), 0.0775890 secs] [Times: user=0.72 sys=0.00, real=0.08 secs] 
2015-12-23T09:49:17.513-0600: 23517.751: [GC 23517.751: [ParNew: 1887488K->209664K(1887488K), 0.0846900 secs] 3411141K->1736344K(8178944K), 0.0852170 secs] [Times: user=0.87 sys=0.00, real=0.08 secs] 
2015-12-23T09:51:25.173-0600: 23645.411: [GC 23645.411: [ParNew: 1887488K->209664K(1887488K), 0.1071930 secs] 3414168K->1750114K(8178944K), 0.1076930 secs] [Times: user=0.98 sys=0.00, real=0.11 secs] 
2015-12-23T09:53:28.136-0600: 23768.374: [GC 23768.374: [ParNew: 1887488K->209664K(1887488K), 0.1078400 secs] 3427938K->1783176K(8178944K), 0.1083380 secs] [Times: user=1.07 sys=0.00, real=0.11 secs] 
2015-12-23T09:53:35.287-0600: 23775.524: [Full GC 23775.525: [CMS: 1573512K->1530268K(6291456K), 6.8972640 secs] 1885246K->1530268K(8178944K), [CMS Perm : 500922K->500814K(1048576K)], 6.8975970 secs] [Times: user=6.88 sys=0.00, real=6.90 secs] 
2015-12-23T09:55:31.262-0600: 23891.500: [GC 23891.500: [ParNew: 1677824K->183059K(1887488K), 0.0303260 secs] 3208092K->1713327K(8178944K), 0.0308480 secs] [Times: user=0.26 sys=0.00, real=0.03 secs] 
2015-12-23T09:55:58.602-0600: 23918.840: [Full GC 23918.840: [CMS: 1530268K->1523387K(6291456K), 6.1626230 secs] 2091458K->1523387K(8178944K), [CMS Perm : 500849K->500817K(1048576K)], 6.1630690 secs] [Times: user=6.16 sys=0.01, real=6.17 secs] 
2015-12-23T09:58:17.040-0600: 24057.278: [GC 24057.278: [ParNew: 1677824K->50679K(1887488K), 0.0279520 secs] 3201211K->1574066K(8178944K), 0.0284640 secs] [Times: user=0.25 sys=0.00, real=0.03 secs] 
2015-12-23T09:58:20.904-0600: 24061.142: [Full GC 24061.142: [CMS: 1523387K->1521374K(6291456K), 6.6551640 secs] 1626323K->1521374K(8178944K), [CMS Perm : 500850K->500822K(1048576K)], 6.6555850 secs] [Times: user=6.65 sys=0.00, real=6.65 secs] 
2015-12-23T10:00:31.863-0600: 24192.101: [GC 24192.102: [ParNew: 1677824K->31375K(1887488K), 0.0253690 secs] 3199198K->1552750K(8178944K), 0.0258880 secs] [Times: user=0.22 sys=0.00, real=0.03 secs] 
2015-12-23T10:00:45.745-0600: 24205.982: [Full GC 24205.983: [CMS: 1521374K->1522622K(6291456K), 5.1653860 secs] 1699115K->1522622K(8178944K), [CMS Perm : 500862K->500827K(1048576K)], 5.1657630 secs] [Times: user=5.15 sys=0.00, real=5.17 secs] 
2015-12-23T10:03:09.224-0600: 24349.462: [Full GC 24349.462: [CMS: 1522622K->1523684K(6291456K), 7.0333450 secs] 2867394K->1523684K(8178944K), [CMS Perm : 500870K->500857K(1048576K)], 7.0338150 secs] [Times: user=7.03 sys=0.00, real=7.03 secs] 
2015-12-23T10:05:31.352-0600: 24491.590: [GC 24491.590: [ParNew: 1677824K->33949K(1887488K), 0.0269380 secs] 3201508K->1557633K(8178944K), 0.0275020 secs] [Times: user=0.21 sys=0.00, real=0.03 secs] 
2015-12-23T10:05:34.666-0600: 24494.904: [Full GC 24494.904: [CMS: 1523684K->1521445K(6291456K), 6.0579700 secs] 1605165K->1521445K(8178944K), [CMS Perm : 500890K->500858K(1048576K)], 6.0583460 secs] [Times: user=6.05 sys=0.01, real=6.06 secs] 
2015-12-23T10:07:45.748-0600: 24625.986: [GC 24625.986: [ParNew: 1677824K->43491K(1887488K), 0.0327170 secs] 3199269K->1564937K(8178944K), 0.0331990 secs] [Times: user=0.23 sys=0.00, real=0.04 secs]

Answer 1

Given the user and real times to perform the collection are the same it looks like the GC has fallen back the serial collector. It does this when there is a concurrent mode failure, eg the tenured space filled up before the concurrent collector could clean up the tenured space. Normally you should see a message stating this so it is likely to be another cause.

However, since you have a very old version of the JVM, I won't rule out upgrading the version of the Java to at least the last release of Java 6, if not upgrading to Java 8.