Full GC after 5 hours of application start and takes up 40 seconds

Question

We process 5 million requests in one run. However, during the run, few of the requests are failing. Upon verifying close, we understood that the Full GC is kicked in and is taking more than 40 sec(to 60 sec) thereby causing timeouts.

Configurations: We have 3 tomcat instances and Apache server. UseParallelGc is used with 20 parallel GC threads.

Here is an excerpt of the GC logs

[GC 9159407K->4631063K(9193536K), 0.1151870 secs]
[Full GC 4631063K->4474675K(9194112K), 40.2139890 secs]
[GC 9005619K->4545178K(9169472K), 0.0426840 secs]
[GC 9051866K->4587006K(9175232K), 0.1107250 secs]
[GC 9093694K->4603255K(9181440K), 0.1011030 secs]
[GC 9106359K->4627576K(9179776K), 0.1338130 secs]
[GC 9130680K->4631971K(9182144K), 0.1962060 secs]
[Full GC 4631971K->234907K(9181568K), 0.7454950 secs]
[GC 4739803K->278254K(9180288K), 0.0559540 secs]
[GC 4783150K->298456K(9181760K), 0.1616380 secs]
[GC 4804632K->300414K(9181376K), 0.0902670 secs]
[GC 4806590K->327319K(9180672K), 0.1219910 secs]
[GC 4832855K->319675K(9149376K), 0.0465920 secs]
[GC 4825211K->339411K(9182720K), 0.1143500 secs]
[GC 4845907K->340064K(9180736K), 0.0491540 secs]

My questions are:

1. Full GC was started only after 5 hours after application start up. Is there any way to tweak it to happen even [edit]sooner without changing the memory sizes of young and old gens?

2. The first full GC run happened(at 16:43:25) for 40 sec but only freed around 0.15 GB. However, another full GC happened within a minute(at 16:44:12) and it freed more than 4 GB. The gap was just a minute. What could be the reason for this?

Please let me know if any other information is needed on this.

Answer 1

-XX:+UseParallelGC uses a parallel collector only for Young generation. It doesn't matter how many threads you have, your Old generation collection always happens in serial mode using only 1 thread if you've only UseParallelGC enabled.

Looking at your logs, I see only a summary of GC which is usually the output of -XX+PrintGC which will not give much clue about the issue. I suggest you to try

-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:-PrintGCTimeStamps

(or) -Xloggc with some location to store ur gc output.

Using the above flags, you can find out more details like memory before and after GC for each generation instead of a one-liner

In your logs, Full GC might be a Major GC which is happening in your Old generation and with just UseParallelGC flag, its always done using a single thread.

After doing some proper analysis, you can use -XX:+UseParallelOldGC which will do the old generation collection in parallel or a CMS or a G1 GC if you're using latest JDK7