High memory usage issues with G1 Garbage collector

Question

We have been testing out G1 garbage collector recently with the following configuration:

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseG1GC -XX:MaxGCPauseMillis=1250 -XX:+PrintTenuringDistribution -Xloggc:${logdir}/gc-$(date +%Y_%m_%d-%H_%M).log -XX:+UseStringDeduplication -XX:+PrintStringDeduplicationStatistics -XX:+PrintPromotionFailure -XX:+PrintAdaptiveSizePolicy -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:ParallelGCThreads=8 -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8M JAVA_OPTS_HEAP: -Xms16g -Xmx16g

We had recently come across an issue recently where two java processes were running with the above configuration on a box with 48 GB RAM and both the processes went on to consume around 20 - 22 GB of RAM each (few small processes consuming the remaining memory), thus filling the entire RAM and then it triggered disk swaps which finally led to the OOM and process getting killed.

This seems worrying because neither the NMT reports this memory usage in a meaningful way nor do we get any clues for this usage from the GC logs. In the NMT stats, the application memory was under 16G and the metaspace usage was under 1G.

We had tried setting maxMetaSpaceSize to 2G but that didn't help either. The RAM usage seems to grow unbounded when the process is running for days.

From the other questions it does seem that G1 garbage collector does tend to consume more memory but the disk swaps is a worrying issue. Could someone please provide some pointers on how this issue could be resolved?

Answer 1

As to long for a comment I put it in as an answer.

A good reading which explains why a java process might consume more memory than -Xmx . Based ony our provided information I believe this would be also the reason in your case.

For G1 there is an OBE Getting Started with the G1 Garbage Collector with details about the function of the G1GC. Have a look there for Recommended Use Cases for G1 . Maybe you would not benefit from using G1.

cited from from the OBE ( O racle B y E xample)

If you are using CMS or ParallelOldGC and your application is not experiencing long garbage collection pauses, it is fine to stay with your current collector.

Answer 2

Here you can find testing results of G1, Parallel, ConcMarkSweep, Serial and Shenandoah garbage collectors in terms of scaling and resource consumption, as well as some suggestions on what settings can be applied to improve results. So you can choose the most appropriate one for your project and reduce memory usage.