当运行 JVM 的 docker 容器达到内存限制时会发生什么？

Question

Let us say, a docker container has JVM running inside it in a server with 16G RAM, 2 core CPU and following are the memory limits on container:

mem_limit: 14G
max_mem: 13G
threads: 1
max_pool_size: 20

max_mem is max heap size used by JVM and mem_limit is max memory used by container. I am trying to understand following things:

1. What happens when JVM reaches it max_mem limit?
2. What happens when container reaches it mem_limit ?
3. Does it restarts with or without OOM errror log messages? Where can we see these logs?
4. How much memory should we leave for container after max_mem ?
5. How does threads effect the memory limit?

Answer 1

1. What happens when JVM reaches it max_mem limit?
It has to crash. If you have a live-restore or on docker-swarm or on kubernetes it will bring up another container swiftly.

2. What happens when container reaches it mem_limit?
It has to report and crash. Answer remains same as above.

3. Does it restarts with or without OOM errror log messages?
It gives OOM error. Question 1, error log is inside the container, so you need to log in into the dead container and see the logs. Question 2, error log is from docker, docker logs must show it. Docker logs will show your container log also.

4. Where can we see these logs?
Answered in above question

5. How much memory should we leave for container after max_mem?
Do a memory model design for your application and add a buffer and limit your container according to that. Make sure you fix your application code if your container keeps crashing because of OOM

6. How does threads effect the memory limit?
Thread is a runner, which uses the resources in your memory or disk. The number of threads choke memory not because they are large in number, but because of the native data in memory they carry along when they are created.

Hope i have answered your questions.

Answer 2

Looking at the docs it seems there is a memswap_limit option so I recon it's safe to assume swap would be used. It could crash cause jvm is good at that however, I imagine it'd just run slow.

The thread itself would not slow down but it would essentially be bottlenecked by slow read/write speed due to swap.

Your max mem question is quite a difficult one to answer without messing about with it a bit. You might find somewhere someone recommends 10-20% headroom but you should really try to find the sweet spot your self.

Note: 14gb ram is a lot for a single container. If possible, you should consider distributing work loads over multiple containers. Docker run allows you to scale containers using --scale arg and orchestration platforms such as kubernetes can manage the process for you.