High Object Copy times resulting in long garbage collection pauses with G1GC

Question

I have a Java app running in a standalone JVM. The app listens for data on one or more sockets, queues the data, and has scheduled threads pulling the data off the queue and persisting it. The data is wide, over 700 data elements per record, though all of the data elements are small Strings, Integers, or Longs.

The app runs smoothly for periods of time, sometimes 30 minutes to an hour, but then we experience one or more long garbage collection pauses. The majority of the pause time is spent in the Object Copy time. The sys time is also high relative to the other collections.

Here is the JVM details:

java version "1.7.0_03" 
Java(TM) SE Runtime Environment (build 1.7.0_03-b04) 
Java HotSpot(TM) 64-Bit Server VM (build 22.1-b02, mixed mode)

Here are the JVM options:

-XX:MaxPermSize=256m -XX:PermSize=256m -Xms3G -Xmx3G -XX:+UseG1GC -XX:-UseGCOverheadLimit 

The process is taskset to 4 cores (all on the same socket), but is barely using 2 of them. All of the processes on this box are pinned to their own cores (with 0 ansd 1 unused). The machine has plenty of free memory (20+G) and top shows the process using 2.5G of RES memory.

Here is some of the gc log output...

[Object Copy (ms): 2090.4 2224.0 2484.0 2160.1 1603.9 2071.2 887.8 1608.1 1992.0 2030.5 1692.5 1583.9 2140.3 1703.0 2174.0 1949.5 1941.1 2190.1 2153.3 1604.1 1930.8 1892.6 1651.9

[Eden: 1017M(1017M)->0B(1016M) Survivors: 7168K->8192K Heap: 1062M(3072M)->47M(3072M)] 

[Times: user=2.24 sys=7.22, real=2.49 secs]

Any ideas on why the Object Copy time and sys time are so high and how to rectify it? There are numerous garbage collections in the log with nearly identical Eden/Survivors/Heap sizes that are only taking 10 or 20 ms.

Answer 1

3gb is not a large heap and the survivor size is also small. Anything else running on those cores? How much garbage are you generating and how often is it collecting? You may want to try it without G1GC as well.

Answer 2

Do you need 3 gigabytes of heap? Garbage-collection pauses get longer as a heap gets bigger, since (though less frequent) there is more work to do when GC is finally needed.

It appears you're locking the heap at 3 gig by setting a minimum. If the app doesn't need 3 gig, this is going to force it to use that much anyway.. and result in a humungous pause when that 3G finally does need to be collected.

I spent quite some time tuning Eclipse IDE for responsiveness, and found very early on that compact heap-sizes had better 'low pause' characteristics than large.

Apart from JVM heap settings, you can make sure your code 'nulls out' data elements & collection items as these are discarded.

This is standard practice in the java.util Collections package, but perhaps you have code that could benefit from it. The 700-wide records could especially be a candidate for this practice, which helps to simplify things for the GC and enable better cleanup from the 'minor GC' sweeps.