Java program is getting slower after running for a while

Question

I have a java program that is a typical machine learning algorithm, updating the values for some parameters by some equations:

for (int iter=0; iter<1000; iter++) {
    // 1. Create many temporary variables and do some computations                         
    // 2. Update the value for the parameters                    
}

The computations of updating parameters are rather complex, and I have to create many temporary objects, but they are not referenced out of the loop. The code in the loop is CPU-intensive, and does not access disk. This program loads a relatively large training dataset, therefore, I granted 10G memory (-Xmx10G) to JVM, which is much larger than it requires (peak at ~6G by "top" command or window's task manager).

I tested it on several linux machines (centos 6, 24G memory) and a window machine (win7, 12G), both with SUN Hotspot JDK/JRE 1.8 installed. I did not specify other JVM parameters except -Xmx. Both machines are dedicated to my program.

On windows, my program runs well: each iteration uses very similar running time. However, the running time on all of the centos machines is weird. It initially runs properly, but slows down dramatically (~10 times slower) at 7th/8th iteration, and then keeps slow down ~10% in each iteration ever after.

I suspect it might be caused by Java's garbage collector. Therefore, I use jconsole to monitor my program. Minor GC happens very frequently on both machines , that is because the program creates many temporary variable in the loop. Furthermore, I used "jstat -gcutil $pid$ 1s" command and captured the statistics:

Centos: https://www.dropbox.com/s/ioz7ai6i1h57eoo/jstat.png?dl=0

Window: https://www.dropbox.com/s/3uxb7ltbx9kpm9l/jstat-winpng.png?dl=0

[Edited] However, the statistics on two kinds of machines differ a lot:

1. "S1” on windows jumps fast between 0 to 50, while stays at "0.00" on centos.
2. "E" on windows changes very rapidly from 0 to 100. As I print the stat for every second, the screenshot does not show its increment to 100. On centos, however, "E" increases rather slowly towards 100, and then reduces to 0, and increases again.

It seems the weird behaviour of my program is due to Java GC? I am new to Java performance monitor and do not have a good idea to optimize GC parameter setting. Do you have any suggestions? Thank you very much!

Answer 1

I'm sorry to post this as an answer but I don't have enough score to comment.

If you think it's a GC related issue I'd change it for the Garbage 1 Collector –XX:+UseG1GC

I found this brief explanation about it: http://blog.takipi.com/garbage-collectors-serial-vs-parallel-vs-cms-vs-the-g1-and-whats-new-in-java-8/

Can you run your software under profiling? Try to use the jprofiler, VisualVM or even the netbeans profiler. It may help you a lot.

I noticed that you have your own encapsulation of a vector and matrix. Maybe your are spending a lot more memory than necessary with that too. But I don't think that is the problem.

Sorry again about not contributing as a comment. (It would be more appropriate)

Answer 2

Giving Java (or any garbage collecting language) too much memory has an adverse effect on performance. The live (referenced) objects become increasing sparse in memory resulting in more frequent fetches from main memory. Note that in the examples you've shown us the faster windows is doing more quick and full GC than Linux - but GC cycles (especially full gcs) are usually bad for performance.

If running the training set does not take an exceptionally long time, then try benchmarking at different memory allocations.

A more radical solution, but one which should have a big impact is to eliminate (or reduce as much as possible) object creation within the loop by recycling objects in pools.

Answer 3

我会考虑在循环外声明变量，因此mem分配完成一次并完全消除GC。

Answer 4

First, it is a common best practice to declare Variables outside of loops to avoid garbace collection. as 'Wagner Tsuchiya' said, try running a profiler if you have doubts about the GC. If you want some tips on GC tuning, i found nice blogpost .

Answer 5

You could try calling System.gc() every couple iterations to see if performance goes up or down. This may help you narrow it down to some of the previous answers diagnostics.

Answer 6

If the GC time is hundreds of milliseconds as shown in your screenshot then GC is likely not the issue here. I suggest you look into lock contention and possibly IO using a profiler (Netbeans is great). I know you stated your program did very little IO but with profiling (much like debugging) you have to remove all your assumptions and go step by step.

Answer 7

In my experience JAVA needs enough memory and 2+ CPU. Otherwise CPU usage will be very extensive when GC starts running.