Optimizing Tomcat / Garbage Collection

Question

Our server has 128GB of RAM and 64 cores, running Tomcat 7.0.30 and Oracle jdk1.6.0_38, on CentOS 6.3.

Every 60 minutes we're seeing garbage collection that was taking 45 - 60 seconds. Adding -XX:-UseConcMarkSweepGC increased page load times by about 10% but got that down to about 3 seconds, which is an acceptable trade-off.

Our config:

-Xms30g
-Xmx30g
-XX:PermSize=8g
-XX:MaxPermSize=8g
-Xss256k
-XX:-UseConcMarkSweepGC

We set the heap at 30 GB to keep 32 bit addressing (I read that above 32 GB the 64 bit addressing takes up more memory, so you have to go to about 48 GB to see improvements).

Using VisualGC I can see that the Eden space is cycling through every 30 - 60 minutes, but not much happens with the Survivor 0, Survivor 1, Old Gen, and Perm Gen.

We have a powerful server. What other optimizations can we make to further decrease the 3 second GC time?

Any recommendations to improve performance or scaling?

Any other output or config info that would help?

Answer 1

It might sound counter-intuitive, but have you tried allocating a lot less memory? Eg do you really need a 30G heap? In case you can get along with 4G or even less: Garbage collection might be more frequent, but when it happens it will be a lot faster. Typically I find this more desirable than allocating a lot of memory, suffering from the time it takes to clean it up.

Even if this will not help you because you really need 30G of memory, others might come along with a similar problem and they might benefit from allocating less.

Answer 2

Seems that you need Incremental GC to reduce pauses:

• -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode

and for tracing without visualgc this always went well for me (output in catalina.out):

• -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps

2013-01-05T22:52:13.954+0100: 15918369.557: [GC 15918369.557: [DefNew: 65793K->227K(98304K), 0.0031220 secs] 235615K->170050K(491520K), 0.0033220 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]

After you can play with this:

• -XX:NewSize=ABC -XX:MaxNewSize=ABC
• -XX:SurvivorRatio=ABC
• -XX:NewRatio=ABC

Reference: Virtual Machine Garbage Collection Tuning

Answer 3

Sorry, no answers only more questions but there is so much pertinent info in this thread, although very old, that perhaps there's some interest in shedding some light on what Tomcat (8.5.29 in my implementation), Spring or the JVM/JRE (jdk_1.8.0_171) may be doing in this case.

I've implemented a Web App that includes Web Services and some background processes (threads) managed by the Spring TaskScheduler (these run constantly.. checking for things to handle principally via a DB). Functionally it all runs fine but I'm in the process of tuning things up and I'm seeing an unacceptable and cyclical CPU consumption on the associated JRE.

Suspecting that my code was inefficient and causing the GC to run unnecessarily (or just tune the GC correctly in any case), I've implemented ggrandes suggestion for tracing GC activity and have taken a close look at the associated documentation.

Once done, I was getting many GC related messages which led me to the next step. As I was debugging the user code, I couldn't make sense of (or correlate with my User Code activity) the GC invocations which are plentiful. So, in debugging mode, I put some breakpoints in the User Code effectively paralyzing ALL User Code related execution and the cyclical GC messages continue at their rhythm as when the User Code is running. The results are that (independently whether the User Code is running or not) every 10-11 seconds the GC is invoked and runs for approx. 11-12 seconds.

Keeping in mind that with the User Code TOTALLY blocked.. this is a sample output..

2019-10-28T13:12:07.358+0100: 6206.987: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 201649K->160689K(264704K), 0.0019574 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:17.675+0100: 6217.304: [GC (Allocation Failure) [PSYoungGen: 41216K->320K(41472K)] 201649K->160761K(264704K), 0.0019036 secs] [Times: user=0.06 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:17.899+0100: 6217.527: [GC (Allocation Failure) [PSYoungGen: 41280K->256K(41472K)] 201721K->160809K(264704K), 0.0019992 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:18.128+0100: 6217.756: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 201769K->160785K(264704K), 0.0022651 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:18.352+0100: 6217.981: [GC (Allocation Failure) [PSYoungGen: 41184K->288K(41472K)] 201745K->160873K(264704K), 0.0020136 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:18.575+0100: 6218.203: [GC (Allocation Failure) [PSYoungGen: 41248K->256K(41472K)] 201833K->160865K(264704K), 0.0019503 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:18.793+0100: 6218.422: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 201825K->160889K(264704K), 0.0019472 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:19.012+0100: 6218.641: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 201849K->160913K(264704K), 0.0019231 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:19.241+0100: 6218.870: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 201873K->160897K(264704K), 0.0022029 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:19.459+0100: 6219.088: [GC (Allocation Failure) [PSYoungGen: 41184K->224K(41472K)] 201857K->160905K(264704K), 0.0024031 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:19.685+0100: 6219.313: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 201865K->160945K(264704K), 0.0018694 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:19.902+0100: 6219.531: [GC (Allocation Failure) [PSYoungGen: 41216K->288K(41472K)] 201905K->160985K(264704K), 0.0019840 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:20.123+0100: 6219.752: [GC (Allocation Failure) [PSYoungGen: 41248K->224K(41472K)] 201945K->160929K(264704K), 0.0021776 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:20.338+0100: 6219.967: [GC (Allocation Failure) [PSYoungGen: 41184K->224K(41472K)] 201889K->160929K(264704K), 0.0018897 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:20.555+0100: 6220.183: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 201889K->160977K(264704K), 0.0019590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:20.771+0100: 6220.399: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 201937K->160953K(264704K), 0.0022302 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:20.991+0100: 6220.619: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 201913K->161001K(264704K), 0.0020969 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:21.229+0100: 6220.858: [GC (Allocation Failure) [PSYoungGen: 41216K->288K(41472K)] 201961K->161041K(264704K), 0.0018507 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:21.445+0100: 6221.074: [GC (Allocation Failure) [PSYoungGen: 41248K->224K(41472K)] 202001K->160985K(264704K), 0.0023642 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:21.659+0100: 6221.288: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 201945K->161017K(264704K), 0.0019729 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:21.875+0100: 6221.504: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 201977K->161025K(264704K), 0.0019721 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:22.109+0100: 6221.738: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 201985K->161033K(264704K), 0.0021257 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:22.338+0100: 6221.966: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 201993K->161041K(264704K), 0.0018824 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:22.565+0100: 6222.194: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 202001K->161017K(264704K), 0.0019543 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:22.785+0100: 6222.414: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 201977K->161049K(264704K), 0.0019685 secs] [Times: user=0.06 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:23.002+0100: 6222.630: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202009K->161049K(264704K), 0.0018725 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:23.240+0100: 6222.869: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 202009K->161025K(264704K), 0.0021566 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:23.451+0100: 6223.080: [GC (Allocation Failure) [PSYoungGen: 41184K->224K(41472K)] 201985K->161025K(264704K), 0.0020730 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:23.682+0100: 6223.311: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 201985K->161057K(264704K), 0.0019011 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:23.933+0100: 6223.561: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 202017K->161025K(264704K), 0.0018953 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:24.185+0100: 6223.813: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 201985K->161065K(264704K), 0.0019484 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:24.419+0100: 6224.048: [GC (Allocation Failure) [PSYoungGen: 41216K->288K(41472K)] 202025K->161097K(264704K), 0.0019002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:24.644+0100: 6224.272: [GC (Allocation Failure) [PSYoungGen: 41248K->256K(41472K)] 202057K->161081K(264704K), 0.0018661 secs] [Times: user=0.06 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:24.872+0100: 6224.501: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202041K->161081K(264704K), 0.0019496 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:25.092+0100: 6224.721: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 202041K->161049K(264704K), 0.0022255 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:25.319+0100: 6224.948: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 202009K->161081K(264704K), 0.0020412 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:25.537+0100: 6225.165: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202041K->161081K(264704K), 0.0019278 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:25.762+0100: 6225.391: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 202041K->161057K(264704K), 0.0020179 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:25.979+0100: 6225.608: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 202017K->161089K(264704K), 0.0018991 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:26.252+0100: 6225.880: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 202049K->161057K(264704K), 0.0018755 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:26.484+0100: 6226.112: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 202017K->161089K(264704K), 0.0019254 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:26.695+0100: 6226.324: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202049K->161089K(264704K), 0.0019129 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:26.918+0100: 6226.547: [GC (Allocation Failure) [PSYoungGen: 41216K->288K(41472K)] 202049K->161121K(264704K), 0.0018535 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:27.156+0100: 6226.785: [GC (Allocation Failure) [PSYoungGen: 41248K->256K(41472K)] 202081K->161097K(264704K), 0.0019399 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:27.387+0100: 6227.016: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202057K->161097K(264704K), 0.0020594 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:27.615+0100: 6227.244: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202057K->161097K(264704K), 0.0021954 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:27.832+0100: 6227.461: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202057K->161097K(264704K), 0.0020552 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:28.047+0100: 6227.676: [GC (Allocation Failure) [PSYoungGen: 41216K->256K(41472K)] 202057K->161097K(264704K), 0.0021104 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:28.266+0100: 6227.895: [GC (Allocation Failure) [PSYoungGen: 41216K->224K(41472K)] 202057K->161073K(264704K), 0.0019352 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:28.474+0100: 6228.103: [GC (Allocation Failure) [PSYoungGen: 41184K->224K(41472K)] 202033K->161073K(264704K), 0.0020573 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:28.690+0100: 6228.319: [GC (Allocation Failure) [PSYoungGen: 41184K->256K(41472K)] 202033K->161105K(264704K), 0.0023471 secs] [Times: user=0.06 sys=0.00, real=0.00 secs] 
2019-10-28T13:12:39.012+0100: 6238.641: [GC (Allocation Failure) [PSYoungGen: 41216K->288K(41472K)] 202065K->161137K(264704K), 0.0020514 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 

So my questions are.. (1) WHAT would be causing these GC invocations, (2) WHY so many and, of course, (3) HOW this can be controlled and/or avoided.. it really takes up an annoying amount of CPU.

Thanks in advance for any input on the matter and sorry for any inconvenience with the thread (moderator, please move this to its own thread if you see fit.. I wasn't sure what to do with it).