Java application slow because of heap

Question

We have serious application issue at peak time application get very very slow and when i check on AppDynamics matrix, my heap memory is full and GC kicked in every minute and that make it very very slow. here is the configuration of my java (tomcat)

OS version is Redhat 5 Linux

java version "1.6.0_05" 64Bit

Java options are -Djava.awt.headless=true -Xmx2048m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC

Major GC collection time spend per min (ms)

CMS Old Gen usage in MB

Par Eden space in MB

Any suggestion why par eden space and old gen hitting hard line?

UPDATE

Here is the last 12 Hour picture of Heap usage and Major GC collection (in Green dots), GC was very high between 3:00AM to 7:00AM but when i restart application around 7:30AM everything is good and application response time was very fast, why reboot fixed everything?

Hurray! 4GB Heap solved problem

Major GC collection time spend per min (ms) after 4GB (Zero Major GC)

CMS Old Gen usage in MB after 4GB Heap

Par Eden space in MB after 4GB Heap

Answer 1

One major GC per minute doesn't at all seem troublesome. Usually it takes about half a second, so that's 1/120th of your overall CPU usage. It is also quite natural that heavy application load results in more memory allocation. Apparently you are allocating some objects that live on for a while (could be caching).

My conclusion: the demonstrated GC behavior is not a proof that there is something wrong with your application's memory allocation.

UPDATE

I have looked more carefully at your diagrams (unfortunately they are quite difficult to read). You don't have one GC per min; you have 60 seconds of major GC per minute, which would mean it's happening all the time. That does look like major trouble; in fact in those conditions you usually get an OOME due to "GC time percentage threshold crossed". Note that the CMS collector which you are using is actually slower than the default one; its advantage is only that it doesn't "stop the world" as much. It may not be the best choice for you. But you do look to either have a memory leak, or a general issue in program design.

Answer 2

When JVM reaches heap maximum size, GC is called more frequently to prevent OOM exception. Normal program execution should not happen at such circumstances. When a new object is allocated and JVM cant get enough of free space, GC is called. This might postpone object allocation process and thus slowdown overall performance. Garbage collection happens not concurrently in this case and you do not benefit from CMS collector. Try to play with CMSInitiatingOccupancyFraction , its default value is about 90%. Setting this parameter to lower values, will force garbage collection before application reaches heap maximum. Thus GC will work in parallel with application not clashing with it. Have a look at article Starting a Concurrent Collection Cycle .

Answer 3

Are you always waiting for GC to take care of removing unused references? Is there some places in your application that you know a reference to a heavy weight object won't be used anymore from that point, but it is not manually nulled? Perhaps setting such heavy weight objects to null manually at the right places could prevent them growing till they hit the end of the heap....