How to run docker daemon on read-only host file system

Question

I have a Raspberry Pi 3 running Ubuntu 16.04.6 LTS on a read only root partition. Now I try to start a docker daemon. But after reboot it says:

ubuntu@z11:~$ docker info
Client:
 Debug Mode: false

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

When trying to restart the docker daemon I get:

ubuntu@z11:~$ sudo systemctl daemon-reload
ubuntu@z11:~$ sudo systemctl restart docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

This is the output of systemctl status docker.service :

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Thu 2016-02-11 17:28:26 CET; 14s ago
     Docs: https://docs.docker.com
  Process: 1620 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 1620 (code=exited, status=1/FAILURE)

Feb 11 17:28:24 z11 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Feb 11 17:28:24 z11 systemd[1]: Failed to start Docker Application Container Engine.
Feb 11 17:28:24 z11 systemd[1]: docker.service: Unit entered failed state.
Feb 11 17:28:24 z11 systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 11 17:28:26 z11 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Feb 11 17:28:26 z11 systemd[1]: Stopped Docker Application Container Engine.
Feb 11 17:28:26 z11 systemd[1]: docker.service: Start request repeated too quickly.
Feb 11 17:28:26 z11 systemd[1]: Failed to start Docker Application Container Engine.
Feb 11 17:28:26 z11 systemd[1]: docker.service: Unit entered failed state.
Feb 11 17:28:26 z11 systemd[1]: docker.service: Failed with result 'start-limit-hit'.

This is my /etc/fstab with a read-only root partition and read-write /var/lib/docker:

proc            /proc                   proc    defaults                                     0 0
/dev/mmcblk0p1  /boot                   vfat    ro                                           0 0
/dev/mmcblk0p2  /                       ext4    noatime,ro                                   0 0
/dev/mmcblk0p3  none                    swap    sw                                           0 0
/dev/mmcblk0p4  /var/lib/docker         ext4    noatime,rw                                   0 0
tmpfs           /var/log                tmpfs   defaults,noatime,nosuid,mode=0755,size=50m   0 0
tmpfs           /var/log/apache2        tmpfs   defaults,noatime,size=10m                    0 0
tmpfs           /var/lib/sudo           tmpfs   defaults,noatime,nosuid,mode=0755,size=2m    0 0
tmpfs           /tmp                    tmpfs   defaults,noatime,mode=1777,size=30m          0 0

When changing the root partition from ro to rw , docker is running nicely after rebooting the system.

How can I get it docker to run with keeping the root partition read-only? Are there other files or directories docker needs to write to?

Answer 1

I want to know what your result of mount looks like?

My situation is a little different from yours. In my case, I got the same issue while I running docker daemon on a root overlayed filesystem. I hope my solution may also help you or anyone else.

According to the guide line, http://docs.docker.jp/engine/userguide/storagedriver/selectadriver.html , docker can not run on a overlay backing filesystem. But fortunately, it can run on a tmpfs which is above the overlay filesystem.

Here is my environment:

• Ubuntu Server 19.10
• Raspberry PI 4B
• Using overlayroot for making the root overlayed on a read-only filesystem

And my solution is:

1. sudo mount -t tmpfs tmpfs /var/lib/docker after boot (edited /etc/fstab may not work for overlayroot)
2. restart docker with sudo systemctl restart docker

If Step2 got failed, just wait for a few seconds and have a retry (Step1 doesn't seem to work immediately).

I also found that if you repeat Step1 for more than two times, Step2 can be executed immediately. Sorry, I cann't explain why.

Answer 2

Obviously, /run (aka /var/run) will be used to open a socket file. Docker is a system daemon and the "docker" command is simply sending the input to the socket file. Basically, you have mounted some subdirectories of /var as rw but not /var/run.

If you wanted to have /var/lib/docker on a persistant storage then it may as well be a better idea to reconfigure the docker.service - probably check your /etc/sysconfig/docker and make it

DOCKER_OPTS="-g /data/docker"

Answer 3

I needed to add this

tmpfs        /var/lib/docker tmpfs   nosuid,nodev         0       0
tmpfs        /var/lib/containerd tmpfs   nosuid,nodev         0       0

to /etc/fstab