Rancher(docker) diskusage cleanup

Question

Rancher system started to use a heavy amount of disksspace. Kubernetes was setup by Rancher's RKE.
Diskusage is already over 5TB however I have only 10-12 replicaset, their real data is binded to PV which uses nfs (which has only a size of 10gb).

df -h --total clearly shows which one takes up so many space:

Filesystem       Size  Used Avail Use%  Mounted on
    overlay      98G   78G   16G  84% /var/lib/docker/overlay2/84db..somehash/merged

I have ~50-60 entry from these.

How can I cleanup these? Is there any maintenance feature in rancher for this? Couldn't find any though.

Answer 1

Kubernetes's garbage collection should be cleaning up your nodes.

This looks a lot an issue I saw with some log collectors like Splunk and Datadog.

If the following usage numbers do not match up. Then using the script below to release the file descriptors.

• df -h /var/lib/docker
• docker system df

Workaround:

ps aux | grep dockerd    <<== This pid
cd /proc/`pid of dockerd`/fd
ls -l |grep var.log.journal |grep deleted.$ |awk '{print $9}' |while read x; do :> $x; done;