How to detect when a Docker container reaches its configured memory limit

Question

Given a Docker container that has been launched with a memory limit , I wonder if it possible to detect when that memory limit is being enforced, and thus the container is about to crash because it's getting out of memory. Is there some signal that I could trap?, or some log file that I can poll? Detecting this after the fact, even with some minutes of delay would still be useful

Answer 1

You can do a script based off these instructions:

# for i in $(docker ps --no-trunc | awk '{print $1}' | grep -v CONTAINER); do \
LIMIT=`cat /sys/fs/cgroup/memory/docker/${i}/memory.limit_in_bytes`; \
USAGE=`systemd-cgtop -b --iterations 1 | grep ${i} | awk '{print $4}'`; \
echo Container ${i} is using $USAGE and its limited at $LIMIT; done;

Container e831b58193287ac61038da29f488423da894d44f810a9b2ff6df63b800d98217 is using 4.5G and its limited at 9223372036854771712
Container 29b484032c895b84e7297091b0b0fc69513c34a03e10ac7906e428538a47e1a1 is using 19.4M and its limited at 9223372036854771712
Container a6ff3e99790f7d3db1da14736e4f48623512fbd46b2683a158aed1736c17073d is using 14.7M and its limited at 9223372036854771712
Container 36db60a74eab475397371bfa0d1ea432486841d036c7cc54e527df1f7b95bbe9 is using 15.3M and its limited at 9223372036854771712
Container 9be4c12074f05eb00e4e5d17eafe733651ff7b7e962faf328e4dd8a10ca8fee7 is using 216.0K and its limited at 9223372036854771712
Container e9da54900df72ab4b1c6233f51b011a68246315d714278e11b54e7b18367593c is using 2.6G and its limited at 9223372036854771712
Container a670124330c56af011d9bc9b86cbd5571c45af9d2d2dbeffdc6d2aee0b04909f is using 350.1M and its limited at 9223372036854771712
Container e2688ff5493c4517ae55e11281ba07c2743f2464da3c08f88fdce1f4bd0c2d8d is using 2.6G and its limited at 9223372036854771712
Container b0b92b056f82f9b4bff5462b6b896d366245dc9af9847d40f85d20b0497c5601 is using 1.1G and its limited at 9223372036854771712
Container 41e9c719c65e5bc568596f4046fab1acd0ac0e15c93c76563ef25ef5c483b357 is using 61.7M and its limited at 9223372036854771712
Container 1ec5305130846edb67218b2010306090d1efed6e6bc12d04de68fae94db330bd is using 155.1M and its limited at 9223372036854771712
Container 1d3cf6ed42eae789ecc01503e704ff29aa4bd9adb0764baf28f393a882bc0c42 is using 366.3M and its limited at 9223372036854771712
Container 4e96919f2a0e5d969ee1907bffe45fed34b7a4e3dbb7a29ab18169579aed7994 is using 68.7M and its limited at 9223372036854771712
Container 31cc3d3a04435c49d7afb8eda5bab0ef0ee309e1dda531fa8e90a012a5543e90 is using 19.0M and its limited at 9223372036854771712
Container 64dd6daaebc1ef852dcb3a0d2294b52c52d92889ab3f6749a99e45cf89eaa7a6 is using 13.0M and its limited at 9223372036854771712
Container 4f33db06c957306ea19a14c22d5248a161fe5d5344c9d4136eb34c13eb76664b is using 14.7M and its limited at 9223372036854771712
Container 85d9a633adc138f6f506df77ec0828078f521fc5cdfa3cc85e20ab43f3ec0979 is using 40.7M and its limited at 9223372036854771712
Container c102bc6691955d1c732a725c3e9b131c3a07c507996076fb64fce3586a46dbb0 is using 114.2M and its limited at 9223372036854771712

Explanation:

1. You get long Ids of your containers
2. You get the cgroup memory.limit_in_bytes for each docker container
3. You use systemd-cgtop utility to get current usage
4. You must do some unit size calculations (because systemd-cgtop returns G/M/K/B and memory.limit_in_bytes is always bytes)
5. Divide the usage by the limit, configure a threshold (ie: 0.8) and trap whatever monitoring system you're using

I don't limit memory on my containers, so it's normal to see 9223372036854771712 as LIMIT. In your case, you'll have it in bytes.

As answered by other users, you can also use docker utilities (not sure which version you're running) which is much easier:

docker stats --no-stream | tr -d "%" | awk '{if($7>10)print "Warning: " $1 "/" $2" is consuming more than 80% of its availabile memory ("$7"%)"}'

Answer 2

One way to monitor this is with docker stats command. You could monitor usage of container even before limit is enforced.

https://docs.docker.com/config/containers/runmetrics/#docker-stats

docker stats container1 container2

You can do some bash scripts too:

how to find MAX memory from docker stats?

Another option is to look at docker events :

https://www.systutorials.com/docs/linux/man/1-docker-events/

Docker containers will report the following events:

attach, commit, copy, create, destroy, detach, die, exec_create, exec_detach, exec_start, export, kill, oom, pause, rename, resize, restart, start, stop, top, unpause, update

Answer 3

If docker kills your container as a result of a memory limit, you will see:

docker inspect --format '{{.State.OOMKilled}}' ${container_id}

set to true.

Note, if the OS kills the process running inside your container before it hits your docker memory limit, all you will see is a SIGKILL received by your app.

---

Proactively, you can monitor docker stats for the container to see if it is approaching the limit as described by other answers here.

---

And in near real time, you can monitor docker events for any container that is killed for OOM with:

docker events --filter type=container --filter event=oom

You can adjust the above events command to monitor specific containers or be time constrained. See the documentation for available flags.