Docker overlay2 folder eating Disk

Question

i have problem with docker overlay2 folder.

When I enter du -sj /var/lib/docker/overlay2 in the time of writing it says:

85G overlay2/

And it keeps growing up. I tried to prune system with command like:

docker system prune -a

But output is:

Total reclaimed space: 0B

I also tried to add max-file and max-size to my docker compose. After adding these lines I recreated containers. But problem is still there, can it be solved in any way, just to stop eating disk space?

EDIT:

docker info:

Server:
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 6
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem:
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local \logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-91-generic
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 62.86GiB
Name: machine
ID: TRX2:AJZG:LNJF:UCPW:MQQB:2PLR:R7KM:VHEV:KBOG:IBT4:JX4R:WR6V
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/\ Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false\

docker image: (for the security reasons i wont show the names)

TAG IMAGE ID CREATED SIZE
latest 667ac029b200 2 years ago 7.6GB
latest cb9df5aac4ac 2 years ago 12.9GB
latest d445c0adc9a5 3 weeks ago 220MB
latest 227d9f0554d3 2 years ago 1.34GB
latest b80092729008 2 years ago 758MB
latest f32a97de94e1 2 years ago 25.8MB

Answer 1

#!/bin/bash
# Remove exited containers
/usr/bin/docker ps -a -q -f status=exited | xargs --no-run-if-empty docker rm -v

# Remove dangling images
/usr/bin/docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi

# Remove unused images
/usr/bin/docker images | awk '/ago/  { print $3}' | xargs --no-run-if-empty docker rmi

# Remove dangling volumes
/usr/bin/docker volume ls -qf dangling=true | xargs --no-run-if-empty docker volume rm

Run with cron. In my case every hour

Answer 2

1. Check with docker volume ls, it will show volumes and size
2. whatever not required clear them
3. If you are using linux based images you can use - apt-get autoremove, to remove orphan packages

Answer 3

You have 10 containers running, and the 2 year old images are a strong indication that you're likely mutating the filesystem inside those containers (writing logs, temp files, maybe even installing apps). A prune will not delete those containers, their images, the filesystem changes within those containers, or the logs of the containers. The part that you'll see in overlay2 are those filesystem changes, so use the following to see what files have been created/changed within each container:

docker container diff ${container_name_or_id}

eg:

$ docker container diff d7c
C /run
A /run/user
A /run/user/1000
C /tmp
A /tmp/.X11-unix
C /home
A /home/user

Your list will likely be longer, and you can go into the container to delete any files created that you no longer need (using docker exec assuming the container has a shell and other CLI tools).

For more on cleaning up a large overlay2 directory, see is it safe to clean docker/overlay2 .