使用 G1GC 时的随机高 Ref Proc

Question

Since my application requires low latency (maximum 10ms), I'm using G1GC with very low Eden (10mb) and low Heap memory (250mb). The goal is to have a lot of minor GC very often but with very small duration. I'm using the following configuration:

-Xms250m \
-Xmx250m \
-XX:+PrintGCDetails -Xloggc:/tmp/myapp-gc.log \
-XX:+PrintGCTimeStamps \
-XX:+PrintGCApplicationStoppedTime \
-XX:+PrintSafepointStatistics \
-XX:+NeverTenure \
-XX:-UseBiasedLocking \
-XX:-UsePerfData \
-XX:+UseG1GC \
-XX:-ResizePLAB \
-XX:-UseStringDeduplication \
-XX:+UnlockExperimentalVMOptions \
-XX:G1NewSizePercent=5 \
-XX:G1MaxNewSizePercent=5 \
-XX:MaxMetaspaceSize=200m \
-XX:ParallelGCThreads=4 \
-XX:ConcGCThreads=1 \
-XX:GCTimeRatio=2 \
-XX:InitiatingHeapOccupancyPercent=50 \
-XX:+DisableExplicitGC \
-XX:+ParallelRefProcEnabled \
-XX:+UseFastAccessorMethods \
-XX:+UseAltSigs \

I executed a test for 1h and it seems that I achieve my goal but for some reasons, I have some Minor GC taking up to 30ms (while the average is on 5ms).

Bellow the statistics for the 1h test:

No.     of GCs  Percentage
0 - 10  9550    99.29%
10 - 20 66      0.69%
20 - 30 2       0.02%
Avg Pause GC Time   4.66 ms

Also a "normal" minor GC and one where the Ref Proc is too high:

11601.845: [GC pause (G1 Evacuation Pause) (young), **0.0035909 secs**]
   [Parallel Time: 2.2 ms, GC Workers: 4]
      [GC Worker Start (ms): Min: 11601844.9, Avg: 11601845.0, Max: 11601845.4, Diff: 0.6]
      [Ext Root Scanning (ms): Min: 0.5, Avg: 1.3, Max: 2.0, Diff: 1.5, Sum: 5.2]
      [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.2]
         [Processed Buffers: Min: 0, Avg: 5.0, Max: 14, Diff: 14, Sum: 20]
      [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 0.4]
      [Termination (ms): Min: 0.0, Avg: 0.5, Max: 0.7, Diff: 0.7, Sum: 1.9]
         [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [GC Worker Total (ms): Min: 1.5, Avg: 1.9, Max: 2.1, Diff: 0.6, Sum: 7.7]
      [GC Worker End (ms): Min: 11601847.0, Avg: 11601847.0, Max: 11601847.0, Diff: 0.0]
   [Code Root Fixup: 0.0 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.1 ms]
   [Other: 1.3 ms]
      [Choose CSet: 0.0 ms]
      [**Ref Proc: 1.0 ms**]
      [Ref Enq: 0.1 ms]
      [Redirty Cards: 0.1 ms]
      [Humongous Register: 0.0 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.0 ms]
   [Eden: 11264.0K(11264.0K)->0.0B(11264.0K) Survivors: 1024.0K->1024.0K Heap: 88152.5K(250.0M)->76944.6K(250.0M)]
 [Times: user=0.01 sys=0.00, real=0.00 secs]
11601.849: Total time for which application threads were stopped: 0.0041718 seconds, Stopping threads took: 0.0000731 seconds 

Problematic one:

6435.887: [GC pause (G1 Evacuation Pause) (young), **0.0356522 secs**]
   [Parallel Time: 4.0 ms, GC Workers: 4]
      [GC Worker Start (ms): Min: 6435886.8, Avg: 6435888.6, Max: 6435890.4, Diff: 3.6]
      [Ext Root Scanning (ms): Min: 0.0, Avg: 1.3, Max: 2.6, Diff: 2.6, Sum: 5.2]
      [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.5, Sum: 0.8]
         [Processed Buffers: Min: 0, Avg: 4.0, Max: 14, Diff: 14, Sum: 16]
      [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 0.8]
      [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 0.6]
         [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [GC Worker Total (ms): Min: 0.0, Avg: 1.9, Max: 3.8, Diff: 3.8, Sum: 7.5]
      [GC Worker End (ms): Min: 6435890.4, Avg: 6435890.5, Max: 6435890.6, Diff: 0.2]
   [Code Root Fixup: 0.0 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.0 ms]
   [Other: 31.6 ms]
      [Choose CSet: 0.0 ms]
      [**Ref Proc: 31.3 ms**]
      [Ref Enq: 0.1 ms]
      [Redirty Cards: 0.1 ms]
      [Humongous Register: 0.0 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.0 ms]
   [Eden: 10240.0K(10240.0K)->0.0B(10240.0K) Survivors: 2048.0K->2048.0K Heap: 82113.0K(250.0M)->71825.0K(250.0M)]
 [Times: user=0.02 sys=0.00, real=0.03 secs]
6435.922: Total time for which application threads were stopped: 0.0362249 seconds, Stopping threads took: 0.0000478 seconds 

Any idea/advice on how I can troubleshoot or overcome this issue?

I'm using java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64 and wildfly-10.1.0. Currently update is not an option.

Answer 1

After analyzing the memory heap, the main problem was related with weak reference types of finalizers.

All of them were having as referent the com.sun.jndi.ldap.LdapSearchEnumeration (Since on our application, we are using LDAP Search and jndi-ldap).

After changing to UnboundID LDAP SDK, problem was solved.

Moreover in case someone is interested, on JDK8 we had the best performance using the Shenandoah GC.