Akka Stream application using more memory than the jvm's heap

Question

Summary:

I have a Java application that uses akka streams that's using more memory than I have specified the jvm to use. The below values are what I have set through the JAVA_OPTS.

• maximum heap size (-Xmx) = 700MB
• metaspace (-XX) = 250MB
• stack size (-Xss) = 1025kb

Using those values and plugging them into the formula below, one would assume the application would be using around 950MB. However that is not the case and it's using over 1.5GB.

Max memory = [-Xmx] + [-XX:MetaspaceSize] + number_of_threads * [-Xss]

Question: Thoughts on how this is possible?

Application overview:

This java application uses alpakka to connect to pubsub and consumes messages. It utilizes akka stream's parallelism where it performs logic on the consumed messages and then it produces those messages to a kafka instance. See the heap dump below. Note, the heap is only 912.9MB so something is taking up 587.1MB and getting the memory usage over 1.5GB

Why is this a problem?
This application is deployed on a kube.netes cluster and the POD has a memory limit specified to 1.5GB. So when the container, where the java application is running, consumes more that 1.5GB the container is killed and restarted.

Answer 1

The short answer is that those do not account for all the memory consumed by the JVM.

Outside of the heap, for instance, memory is allocated for:

• compressed class space (governed by the MaxMetaspaceSize )
• direct byte buffers (especially if your application performs.network I/O and cares about performance, it's virtually certain to make somewhat heavy use of those)
• threads ( each thread has a stack governed by -Xss ... note that if mixing different concurrency models, each model will tend to allocate its own threads and not necessarily provide a means to share threads)
• if native code is involved (eg perhaps in the library Alpakka is using to interact with pubsub?), that can allocate arbitrary amounts of memory outside of the heap)
• the code cache (typically 48MB)
• the garbage collector's state (will vary based on the GC in use, including the presence of any tunable options)
• various other things that generally aren't going to be that large

In my experience you're generally fairly safe with a heap that's at most (pod memory limit minus 1 GB), but if you're performing exceptionally large I/Os etc. you can pretty easily get OOM even then.

Your JVM may ship with support for native memory tracking which can shed light on at least some of that non-heap consumption: most of these allocations tend to happen soon after the application is fully loaded, so running with a much higher resource limit and then stopping (eg via SIGTERM with enough time to allow it to save results) should give you an idea of what you're dealing with.