flink on docker disk full

Question

I created a very small cluster in docker (1 job manager 3 task manager). I have 2 jobs (1 streaming and 1 batch which is restarted when it's finished).

I have 58G free space (approx. 30G used). When I start the aggregator, the job manager started to eat the disk. After some hour my disk is full. But when I check the FS ( du -su /*/ ), it only contains about 30G of files.

I do not understand why it's happened, and where is the data. I try to check inside the jobmanager container but no success.

Any idea?

Answer 1

I had the same problem, it appears it was the .out file in the /usr/local/flink-{1-0.3}/log folder which was over 6GB. I had to delete the file and redeploy the jobs.

Answer 2

Are you using Docker Meta for Mac? I had a "full disk space" issue and I couldn't build my projects anymore. The steps that worked for me:

1) df -h --> check that the host has enough available space

If that's the case, do the following:

1) docker exec -it your-container bash --> connect to your container

2) df -h /var/www --> check disk space

If there's no more available space, exit the container and do the following:

1) docker rm $(docker ps -aq) --> stop and delete all containers

2) docker rmi --> delete all images

3) ls -lh ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 --> check this file size and delete it

4) quit Docker and reopen it

5) quit the terminal and reopen it

6) Build again