Docker container lost file after restart docker service or reboot

Question

My server's main drive nearly full. So I move /var/lib/docker directory to second drive location /media/my-username/sec-drive/docker using

sudo -s # enter root mode
service docker stop
rsync -aXS /media/my-username/sec-drive/docker /var/lib/docker
rm -rf /var/lib/docker
ln -s /media/my-username/sec-drive/docker /var/lib/docker
serivce docker start

then I start my all docker container by using docker-compose up -d all containers works just fine.

But when I reboot or restart docker service, one of my containers lost a bunch of files(other containsers works just fine). One of those files is libmxnet.so (filemode:777) under /opt/myproj/mxnet/

use local mxnet
RuntimeError: Cannot find the files.
List of candidates:
/opt/myproj/mxnet/libmxnet.so
/opt/myproj/mxnet/libmxnet.so
/opt/myproj/mxnet/../../build/libmxnet.so
/usr/local/nvidia/lib/libmxnet.so
/usr/local/nvidia/lib64/libmxnet.so
../../../libmxnet.so

Those files seems lost randomly. In mxnet folder __init__.py lost but __init__.pyc stays fine. That's really wired.

Then I try to remove images and containers and import again, just turn out same result.

UPDATE:

This error occurred on another server again. But this time I've reinstalled the system and haven't move docker to another drive. Seems it has nothing to do with docker directory location

Answer 1

You have to instruct the docker daemon that you change the folder.

In your docker.service you should add a parameter (-g):

FROM:
ExecStart=/usr/bin/docker daemon 
TO:
ExecStart=/usr/bin/docker daemon -g /new/path/docker

Some references here:

https://www.rb-associates.co.uk/blog/move-var-lib-docker-to-another-directory/

https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux