交换内存连续性增加

Question

I have a web application running in glassfish in RHEL. For the application, these are set:

Heap Memory:4GB
Perm Gen:1GB

JConsole shows:

heap memory - 500mb
non heap memory - 350mb 
threads =378

Top shows:

PID   User       PR  NI  VIRT RES  SHR S  %CPU  %MEM   TIME+   COMMAND
17948 root       20   0 12.8g 1.9g  22m S  1.5   16.0  14:09.11 java

From starting itself process is consuming 12.8G.

Top also shows:

Mem:  12251392k total, 11915584k used,   335808k free,    47104k buffers
Swap:  8322944k total,  6747456k used,  1575488k free,   177088k cached

The problem is swap space is continuosly increasing. When no swap space is left, the web application stops respondng.

1. Killing process does not reduces used swap space but only after computer reboot. Why?
2. Why is the process consuming 12.8 GB of virtual space when started?
3. How to approach to resolve this issue?

Update:

The jconsole output(recorded for 24 hours) shows that heap memory and non heap memory didn't increase much. Even though the swap space redcued by 1.5Gb in the same period: Jconsole output

Answer 1

You could have a look at these answers to get an impression of the meaning of top's output. You can use a script to roughly report what uses your swap space.

To my knowledge Linux' swap system is not that straightforward. The kernel first swaps out inactive memory, probably other application's memory, to give GF enough resources. This will not be instantly swapped back in when GF is terminated. You could try to swapoff -a to force Linux to swap things back in, but remember to re-enable it via swapon -a .

The VIRT space due to top's manpage:

The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out and pages that have been mapped but not used.

I doubt that the OS reports on memory usage are that good in order to debug your Java application. You should have a look into your JVM's memory with tools like JVisualVM (part of Oracle's JDK). Observe the progress of memory usage for a relevant period of time.

Further you can try to analyze a heap dump with a tool like the Eclipse Memory Analyzer (MAT). MAT has some nice reports that can help to find memory leaks. If your application's memory usage constantly grows it seems to have a leak. Otherwise it would simply have not enough memory available.