Java - Internal Native Memory usage is too high

Question

What is the Internal Native Memory in Java, and how can i limit it? I'am using the one of popular java application, like tomcat, but i have problem with very high memory usage, after few hours.

I try to detect what is the problem, and i checked the JCMD:

Total: reserved=20207MB, committed=18968MB
-                 Java Heap (reserved=10000MB, committed=10000MB)
                            (mmap: reserved=10000MB, committed=10000MB)

-                     Class (reserved=1069MB, committed=50MB)
                            (classes #6159)
                            (malloc=1MB #19368)
                            (mmap: reserved=1068MB, committed=48MB)

-                    Thread (reserved=570MB, committed=570MB)
                            (thread #2039)
                            (stack: reserved=561MB, committed=561MB)
                            (malloc=7MB #10216)
                            (arena=3MB #4076)

-                      Code (reserved=252MB, committed=52MB)
                            (malloc=8MB #10918)
                            (mmap: reserved=244MB, committed=44MB)

-                        GC (reserved=553MB, committed=553MB)
                            (malloc=150MB #129854)
                            (mmap: reserved=403MB, committed=403MB)

-                  Compiler (reserved=2MB, committed=2MB)
                            (malloc=2MB #2346)

-                  Internal (reserved=7726MB, committed=7726MB)
                            (malloc=7726MB #67913)

-                    Symbol (reserved=10MB, committed=10MB)
                            (malloc=8MB #67876)
                            (arena=2MB #1)

-    Native Memory Tracking (reserved=5MB, committed=5MB)
                            (malloc=1MB #7514)
                            (tracking overhead=5MB)

-                   Unknown (reserved=20MB, committed=0MB)
                            (mmap: reserved=20MB, committed=0MB)

At the begening memory usage is like this:

Total: reserved=14324MB, committed=12096MB
-                 Java Heap (reserved=10000MB, committed=9050MB)
                            (mmap: reserved=10000MB, committed=9050MB)

-                     Class (reserved=1069MB, committed=49MB)
                            (classes #6130)
                            (malloc=1MB #16522)
                            (mmap: reserved=1068MB, committed=48MB)

-                    Thread (reserved=400MB, committed=400MB)
                            (thread #1381)
                            (stack: reserved=394MB, committed=394MB)
                            (malloc=4MB #6926)
                            (arena=2MB #2760)

-                      Code (reserved=251MB, committed=48MB)
                            (malloc=8MB #10237)
                            (mmap: reserved=244MB, committed=41MB)

-                        GC (reserved=501MB, committed=466MB)
                            (malloc=98MB #103127)
                            (mmap: reserved=403MB, committed=368MB)

-                  Compiler (reserved=1MB, committed=1MB)
                            (malloc=1MB #1142)

-                  Internal (reserved=2068MB, committed=2068MB)
                            (malloc=2068MB #49248)

-                    Symbol (reserved=9MB, committed=9MB)
                            (malloc=7MB #67619)
                            (arena=2MB #1)

-    Native Memory Tracking (reserved=5MB, committed=5MB)
                            (tracking overhead=4MB)

-                   Unknown (reserved=20MB, committed=0MB)
                            (mmap: reserved=20MB, committed=0MB)

i thought that maybe it is memory leak in application, but then it would probably high usage is in Heap size. In addition, i found few articles about bug in glibc, so i've added to systemctl of app

Environment="MALLOC_ARENA_MAX=2"

Also, i've try 4.

My GLIBC version is: ldd (Debian GLIBC 2.24-11+deb9u3) 2.24

I'am using server Debian 9 x64 (updated) with 24 threads, 24GB of RAM. Of course is dedicated only for this software, and there is no any other applications. I made heapdump, (when usage is ~22gb) and in Overview of Eclipse memory analyzer i see:

Size: 2,3 GB Classes: 6,3k Objects: 2,3m Class Loader: 23

Do you have any idea, how check what is inside this internal memory, or how can i limit usage? My application is all time stucked, when memory is over, but java process is still running. Just not working, but process still exist.

Answer 1

Are you running on Linux? If so, grab a live core of your program (using gcore) and look at that core with https://github.com/vmware/chap

The following commands are likely to be useful:

count writable
count rxonly
count readonly
count inaccessible
summarize writable
summarize rxonly
summarize readonly
summarize inaccessible

Given that you are using a largish number of threads, you should also try:

count stacks

That will show you how much memory is being used for stacks actively associated with threads (as opposed to cached).

If the number is high, you can determine how much of those stacks are generally wasted by using:

redirect on
describe stacks

If the stacks are much larger than needed, you can trim them by using "ulimit -s" to set the maximum stack size (in some script that starts the process).

If you want to understand memory used by native code, try:

count allocated
count used
count free

(or summarize/list/show in stead of count, but generally after "redirect on")