Java heap dump and the heap size after the heap analysis differs

Question

I am experiencing memory leak and here goes some detail.

At the time of after-leak,

• top shows 50GB memory as residential
• heap dump file size is 25GB
• eclipse MAT analyzer tells me the heap size is 10GB

At the time of before-leak,

• top shows 30GB memory as residential
• heap dump file size is 20GB
• eclipse MAT analyzer tells me the heap size is 10GB

I am pretty surprised that the difference between top, heap-dump size, and the actual heap size. I am guessing that the difference between top and heap is the possibility of garbage collector heap and native heap areas. But, how come the heap dump file size and the actual heap size (from eclipse MAT analyzer) could differ?

Any insight on this problem?

UPDATE / ANSWER

Some of suggestions are to use jcmd ( https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html ) as the website tells "Native Memory Tracking". But, if you read the page carefully, you will see

Since NMT doesn't track memory allocations by non-JVM code, you may have to use tools supported by the operating system to detect memory leaks in native code.

So, in case of the leak inside the native library, jcmd is not an option.

After crawling the Internets for days and trying out various profilers, most effective for this problem is using jemalloc profiler.

This page helped me a lot! https://gdstechnology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/

Answer 1

top and other OS level tools show how much system memory does your JVM process consume. Java heap, defined by -Xmx command line option, is only a part of that memory. Apart from heap JVM needs some memory for itself. Then there are java threads, each requiring a certain amount of memory. And Metaspace/Permanent Generation. And several others. You can read this blog post and this SO answer for more information.

About the size of the dump file and the actual heap size the answer of @arnab-biswas is certainly true. MAT reports the size of actually used heap, consumed by live objects. But heap dump contains the whole of the heap, including garbage.

Answer 2

I have experienced similar situation. The difference (HPROF file size - Size of the heap indicated by MAT) is effectively garbage (unreachable objects). Unreachable object Histogram in MAT should help here.

jmap -F -dump:live,format=b,file=<file_name.hprof> <process_id> will only dump live objects and NOT garbages.

Answer 3

In order to monitor the native memory you need to start your application with -XX:NativeMemoryTracking=summary or -XX:NativeMemoryTracking=detail . Note that there is a performance penalty, so think twice before doing it in production.

When memory tracking is active you can use jcmd <pid> VM.native_memory summary . There are other commands available as well, check https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html or search for native memory tracking.

EDIT: I didn't follow the links before answering, you may be looking for something like https://github.com/jeffgriffith/native-jvm-leaks instead.

Answer 4

You are asking for an answer drawing from credible/official sources. Let me give it a try.

1) why is the memory consumed by my JVM process (shown by Top) larger than the heap size?

Because the total memory consumption of the JVM process consists of more things than just the Java heap. A few examples:

• Generated (JIT:ed) code
• Loaded libraries (including jar and class files)
• Control structures for the java heap
• Thread Stacks
• User native memory (malloc:ed in JNI)

Credible/official sources: Run-Time Data Areas and this blog post

2) why is the heap dump size much bigger than what MAT reports?

Because MAT does not show the complete heap. During the index creation, the Memory Analyzer removes unreachable objects because the various garbage collector algorithms tend to leave some garbage behind.

Credible/official sources: MemoryAnalyzer/FAQ

Answer 5

Heap dump : A heap dump is a snapshot of the memory of a Java process at a certain point of time. There are different formats for persisting this data, and depending on the format it may contain different pieces of information, but in general the snapshot contains information about the java objects and classes in the heap at the moment the snapshot was triggered. Usually a full GC is triggered before the heap dump is written so it contains information about the remaining objects.

For information related to MAT found here http://help.eclipse.org/neon/index.jsp?topic=/org.eclipse.mat.ui.help/welcome.html