Docker container keeps growing

Question

I have a python script that on a loop

1. Downloads video chunks from AWS S3 to /filename.
2. Sorts files in order and concatenates them.
3. Uploads entire processed video file to AWS S3.
4. Deletes folder /filename.

Then continues on a loop until the AWS SQS queue is empty.

Script works great! I have run it for months. The hard drive space varies but never gets about 5%, depending on size of the video.

I decided to put this script in a docker container and run docker-compose so I could run a bunch of them at a time.

The problem is the hard drive fills up! I know with 5 running the space on the disk will be higher, but when I'm done processing, the file gets deleted.

But with docker, it seems to be a cache or something. I exec into each container and they are running fine. Deleteing old files and all.

No clue what the difference between, in a docker container and running as a service would have as impact on the HD.

To add to this, when I "rm" the docker containers, the hard drive space frees up. I run a "docker ps -s" and the space on the containers is not crazy. Just seems like when you "rm" a file inside the docker container it never really "rm"s it.

Answer 1

If you're downloading the image to a directory NOT volumed mapped from the host, the docker container will not release the used disk space until the container is removed--anything done in the container is ephemeral, but the HOST doesn't know the state of what's going on inside the container.

In this sense it's a lot like a virtual machine image, backed by a file that just grows as needed, but never shrinks. Docker has a directory for a running container tracking changes. On the host you can find the files backing the running container in /var/lib/docker/containers/<id>

If you need your containers to share disk space, I'd recommend you map a shared volume from the host into each docker container images to share.

Try the following

 docker run -ti -v /host/dir:/container/dir ubuntu bash

The above would run the ubuntu image in terminal interactive mode and mounting the host's directory /host/dir inside the running container. Anything the container writes to /container/dir will appear in the hosts /host/dir and any other containers mounting it will see the changes as well.

Just remember anything done in the shared volume is seen by all containers that mount it, so be careful when adding and deleting files/directories from it!

Answer 2

I would suggest you to use volumes, and mount these volumes in your containers. Changes on volumes are instantaneous, as opposed to changes made to the containers filesystem (which is not really removed until you delete the container).

Have a look at the docs here

Answer 3

Two things to look at:

1. The container's read/write filesystem layer. You can get an idea for what is happening here with docker diff on the specific container id to see all the filesystem changes made by your container.

2. The container logs. You can view them with docker logs . On a long running container, these may become excessive. See this answer for the process to limit the log size on one container or to make those limits the defaults for all newly created containers.

Lastly, if you are running docker inside a VM, eg Docker Desktop, then you may be seeing filesystem changes expanding the copy-on-write VM filesystem which isn't easy to undo. (The underlying device doesn't know the OS isn't using the bytes anymore, so once the zero'd bytes have been replaced by data, even if the FS no longer points to those bytes, they are assumed in use.) However, since you indicated removing the container frees the space, this is unlikely to be your issue.

It's also unlikely to be solved by a volume, since your application makes the same filesystem changes to the volume that it would to the read-write layer of the container. All it would do is persist those changes between containers.