Gitlab runner: cannot reclaim Docker space on Ubuntu - no space left on device

Question

I am running a private Gitlab Group runner on EC2 (Ubuntu 18.04). It's recently and frequently started failing build jobs at various stages, but all with the same root cause: no space left on device .

On logging in to the EC2 instance, I can see

  System load:                    0.0
  Usage of /:                     99.5% of 29.02GB
  Memory usage:                   14%       
  Processes:                      109

  => / is using 99.5% of 29.02GB
  => There are 3 zombie processes.

Disk free space shows / and /var/lib/docker/overlay2 at 100% usage:

/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           395M  928K  394M   1% /run
/dev/xvda1       30G   29G  140M 100% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
overlay          30G   29G  140M 100% /var/lib/docker/overlay2/ed591...60f1/merged
shm              64M     0   64M   0% /var/lib/docker/containers/e9de...f8ed/mounts/shm
overlay          30G   29G  140M 100% /var/lib/docker/overlay2/0956c...e51f/merged
shm              64M     0   64M   0% /var/lib/docker/containers/4cab...0ba8/mounts/shm
/dev/loop1       18M   18M     0 100% /snap/amazon-ssm-agent/1566
/dev/loop3       29M   29M     0 100% /snap/amazon-ssm-agent/2012
/dev/loop2       97M   97M     0 100% /snap/core/9436
/dev/loop4       97M   97M     0 100% /snap/core/9665
tmpfs           395M     0  395M   0% /run/user/1000

Docker disk usage shows ~21GB, apparently unclaimable:

/# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              2                   2                   746MB               0B (0%)
Containers          2                   2                   8.989MB             0B (0%)
Local Volumes       3                   3                   20.4GB              0B (0%)
Build Cache         0                   0                   0B                  0B

Pruning does nothing:

/# docker system prune
Total reclaimed space: 0B

How can I identify what is using this disk space and ultimately reclaim it?

Answer 1

Executing docker system prune with default options reclaims only space reserved by idle containers, networks - it does not remove images.

In order to remove all unused images too, you need to run:

docker system prune -a