Glassfish web app running slow

Question

I have a web app running in glassfish 3.1.2. The web app is running slow. I am not able to understand the reason. In jvm options: -Xmx8192m, -XX:MaxPermSize=3072m.

The top command is giving:

Mem:  16269852k total, 16188212k used,    81640k free,   202544k buffers
Swap:  8208376k total,   937308k used,  7271068k free,  5715516k cached



  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                    
13037 root      20   0 13.5g 8.4g  65m S 114.7 54.3   9899:38 java                                                                                                      
13495 root      20   0 15288 1448  900 R 41.5  0.0 489:51.24 top                                                                                                        
29747 root      20   0  197m 2544 1524 R 36.6  0.0 449:20.08 mysql                                                                                                      
 8390 root      20   0 1693m 235m  20m S  3.3  1.5   1:12.61 java                                                                                                       
24987 user1     20   0  151m  52m 7092 S  1.3  0.3  38:13.69 Xvnc                                                                                                       
   12 root      20   0     0    0    0 S  1.0  0.0  17:31.10 events/1                                                                                                   
 7386 user1     20   0  304m  12m 7040 S  0.7  0.1   0:23.11 gnome-terminal                                                                                             
 8705 root      20   0 15276 1472  952 R  0.7  0.0   0:05.95 top                                                                                                        
    4 root      20   0     0    0    0 S  0.3  0.0   2:10.46 ksoftirqd/0                                                                                                
 1933 root      20   0 64116  624  524 S  0.3  0.0   5:27.69 sshd                                                                                                       
31445 mysql     20   0 1862m 335m 6280 S  0.3  2.1   4213:09 mysqld                                                                                                     
    1 root      20   0 19352 1208  996 S  0.0  0.0   1:05.27 init                                                                                                       
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.09 kthreadd   

free -g
             total       used       free     shared    buffers     cached
Mem:            15         15          0          0          0          5
-/+ buffers/cache:          9          5
Swap:            7          0          6

jvisualvm screenshots:

What can be the possible reason of slowness?
Why is gc consuming cpu even though the allocated max heap and max perm gen space is higher? 
Why is 99.5% RAM being used although the processes in top are not consuming much space? 
After restarting glassfish it runs fast, the ram consumption is slow.

Thanks,

Answer 1

Your max permgen size is much too big. In the screenshots, it's only using just above 300MB, so setting the max size to be just over 3GB won't really do much. You can explicitly set the perm gen size you want, but it's not a good idea.

Secondly, your heap size looks to be too big. You never reach 5GB in that screenshot, even though the max size is 8GB. It is better not to size the heap to be bigger than you need. If you get problems with a leaky app, or just heap exhaustion, you'll find that the impact of it is much more severe. When you do set the size of your heap, set both Xmx (max) and Xms (min) to be the same value, which will stop the heap from resizing.

A better view of this would be in the Visual GC tab. It looks to me like the garbage collection you're seeing is just young GC, which shouldn't be concern. When you're thinking of GC in the context of app slowness, the thing to look at first is pause times - how long each full GC is taking, since they will pause your app.

Possible reasons for slowness in your app could be any one (or more) of a number of things. It could be due to resource contention, for instance - maybe your JDBC connection pool isn't big enough, or your app is leaking connections.

As for the memory usage in Top, do be aware that you have sorted by the %CPU column, not the %Mem column, so that would be the first thing to look at there.