Hazelcast (JVM) memory 设置

Question

I have installed a cluster of 3 hazelcast nodes. Each hazelcast node is installed in a dedicated VM (running with centos 7). Each VM has 16GB RAM and a 15GB HDD. for each hazelcast node I set the heap memory size in the start.sh script as MIN HEAP SIZE=8G and MAX HEAP SIZE=10G.

However, when I monitor the memory usage on the VM, it is not matching the heap memory usage of hazelcast at all. Let me give you an example: the VM is showing the memory usage 80%. however when I check hazelcast heap memory usage it is at 45%. it is like the VM memory usage is following the higher memory usage of hazelcast. then when the GC starts in hazelcast the heap memory usage on hazelcast decreases but the VM memory usage never decreases. in the VM I also disable THP and set the swappiness to 10 as mentioned in the performance tuning documentation Performance Tuning (hazelcast.com)

So is there a missing configuration that I should do for hazelcast? What can/should I do?

Answer 1

it is like the VM memory usage is following the higher memory usage of hazelcast. then when the GC starts in hazelcast the heap memory usage on hazelcast decreases but the VM memory usage never decreases.

This is typical behaviour for any JVM application, it is not specific to Hazelcast. By default JVM rarely returns the memory to OS, but it is possible to configure it that way.

See eg this answer for more details on behaviours of particular JVMs.

You say you are using a dedicated VM for Hazelcast. In such scenario, it doesn't make sense to return the memory to OS and it is best to use the same MIN and MAX heap size so there is no heap resizing when Hazelcast is running.