Docker Error creating metadata file: no space left on device

Question

I have the following disk space utilization:

Filesystem                                            Size  Used Avail Use% Mounted on
tmpfs                                                  32G  2.7M   32G   1% /run
tmpfs                                                 1.0M     0  1.0M   0% /union
tmpfs                                                 1.0G  5.7M 1019M   1% /union/rw
/dev/ram0                                             227M  227M     0 100% /union/image
unionfs                                               1.0G  5.7M 1019M   1% /
devtmpfs                                               32G     0   32G   0% /dev
/dev/mapper/vgroot-var                                 12G   11G  1.4G  89% /union/var
/dev/mapper/vgroot-local                              708G  457G  252G  65% /SYS/localstorage
/dev/mapper/vgroot-scratch                            165G  1.3G  164G   1% /tmp
tmpfs                                                  32G     0   32G   0% /dev/shm
tmpfs                                                 5.0M     0  5.0M   0% /run/lock
tmpfs                                                  32G     0   32G   0% /sys/fs/cgroup
192.168.52.254:/srv/ql-common                          14G  4.9G  8.2G  38% /etc/qlustar/common
192.168.52.254:/srv/apps/chroots/focal-base-gpu-gsc3   79G   47G   29G  63% /union/nfs
tmpfs                                                 6.3G     0  6.3G   0% /run/user/0

My problem is: sometimes, when I want to run a new docker image, I receive the following error:

Docker Error creating metadata file: no space left on device

The only way that I could solve the error is by doing the following procedure:

1- Stop docker service
2- empty /var/lib/docker directory
3- reboot machine

However, the same error would come back later. I tried all the other solutions such as docker prune, docker rm, docker stop, docker rmi that are discussed here and here but the abovementioned error keeps showing.

Is there any way to redirect /var/lib/docker to use another directory with more free space? or is there any other workaround to this problem?

Answer 1

One work around can be check disk utilisation by docker images by running docker images .

Identify unwanted images and perform cleanup - this way you can reclaim certain disk space.

If you need more space then you can add separate disk or partition existing disk and then mount it to /var/lib/docker

Note: What ever changes that are being suggested for extending disk partition for /var/lib/docker are purely experimental, make sure that you are not performing these steps directly on production environment - You can take this to production only when you had done all necessary testing and confident enough to go ahead with production environment.

If you are planning to mount /var/lib/docker on separate disk or partition then you need to perform below steps:

1. Stop docker containers and stop docker daemon.
2. Take backup of /var/lib/docker along with file permission preserved and store it in different location wherever you have enough space (for example /opt/)
3. Remove files at /var/lib/docker
4. Mount your new disk partition to /var/lib/docker
5. Restore the backup that you have taken at step #2 and restore it on /var/lib/docker
6. Start your docker daemon and start your containers.

Now, /var/lib/docker is mounted on separate disk

Answer 2

sudo systemctl stop docker
sudo mv /var/lib/docker/ /SYS/localstorage/docker/
sudo ln -s /SYS/localstorage/docker/ /var/lib/docker
sudo systemctl start docker