Docker host & no space left on device

Question

I'am using Rancher to manage some EC2 hosts (4 nodes in an auto-scaling group) & to orchestrate containers. Everything works fine.

But, at some point, I have a recurrent problem of disk space, even if I remove unused and untagged images with this command

docker images --quiet --filter=dangling=true | xargs --no-run-if-empty docker rmi

Like I said, even if I run this command above, my hosts are continuoulsy running out of space :

Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G   12K  7.9G   1% /dev
tmpfs           1.6G  1.4M  1.6G   1% /run
/dev/xvda1       79G   77G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            7.9G  7.5M  7.9G   1% /run/shm
none            100M     0  100M   0% /run/user

I'am using rancher 1.1.4 and my hosts are running Docker 1.12.5 under Ubuntu 14.04.4. LTS.

Is there something I miss? What are the best practices to configure docker for production hosts in order to avoid this problem?

Thank you for your help.

Answer 1

Do you use volumes mounts ( docker run -v /local/path:/container/path ) for persistent data of your containers ?

If no, data written by your containers (database, logs ...) will always grow the last layer of your image run.

To see the real size of your current running containers :

docker ps -s

You can also use tools such as https://www.diskreport.net to analyse your disk space and see what has grown between two measures.