No space left on device due to docker/cron

Question

I have a basic server on google cloud that just runs a docker container via cron once every 30 minutes. I noticed that the docker command stopped working and I got an error saying

docker: Error response from daemon: no space left on device.

I then noticed that I got this error even when I trying to autocomplete in bash by typing cd path/ and hitting tab . I figured out something was probably wrong with the storage so I tried df -h and it showed this:

Filesystem      Size  Used Avail Use% Mounted on
udev            860M     0  860M   0% /dev
tmpfs           175M   19M  157M  11% /run
/dev/sda1       9.7G  9.7G     0 100% /        
tmpfs           871M     0  871M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           871M     0  871M   0% /sys/fs/cgroup
tmpfs           175M     0  175M   0% /run/user/1001

As you can see /dev/sda1 is 100% full for some reason. Why is this happening and how can I fix it?

I noticed there were several thousand exited docker containers so I removed this with this command:

docker rm -v $(docker ps -a -q -f status=exited)

Now the storage usage is 61%, which is still too high.

Answer 1

When running containers that you do not need to keep in a stopped state, it's a good practice to use docker run --rm ... to automatically cleanup the stopped container.

If the container generates volumes, visible in docker volume ls , you'll likely want to clean these once the data is no longer needed. with the --rm flag on run, anonymous volumes (which appear as a long unique id) will be automatically deleted.

Docker also provides the command:

docker system prune

which you can automate to cleanup images, containers, networks, and even volumes. Note that you should take time to understand what this command does before running it, and especially before automating it. When scripting this command, you can use the -f flag to bypass the prompt.

Before running a prune, you can check:

docker system df

to see how much disk is being used by each component, and each component has it's own prune command, eg docker container prune and docker volume prune , should you want to clean just one area.

For more details on the prune command, see: https://docs.docker.com/engine/reference/commandline/system_prune/