Docker Swarm shared volumes

Question

Currently I am building a Docker Swarm cluster. It consists of 3 managers and 3 workers. The app I will be deploying on this setup consists of a laravel backend that needs its code in multiple containers for scalability. I already tried GlusterFS volumes and rex-ray with a Ceph backend as shared storage for my volumes. GlusterFS is not reliable and Ceph is in my case a bit overkill and too damn difficult :)

The current setup looks something like this. I have a Percona cluster outside of docker, I run GlusterFS on these servers and simply mount them into the Docker Workers.

                                                Docker Managers
+-------------------------------------------------------------+
|                                                             |
|   +---------+   +---------+   +---------+    +---------+    |
|   |         |   |         |   |         |    |         |    |
|   | HAproxy +---+ HAproxy +---+ HAproxy +----+   SSL   |    |
|   |         |   |         |   |         |    | Manager |    |
|   +----+----+   +----+----+   +----+----+    +---------+    |
|        |             |             |                        |
+-------------------------------------------------------------+
         |             |             |
         |             |             |           Docker Workers
+-------------------------------------------------------------+
|        |             |             |                        |
|   +----+-------------+-------------+--------------------+   |
|   |                                                     |   |
|   |                      Applicaties                    |   |
|   |                                                     |   |
|   +---+--------------+---------------+--------------+---+   |
|       |              |               |              |       |
|       |              |               |              |       |
|   +---+----+     +---+----+     +----+---+     +----+---+   |
|   | Mysql  |     | Mysql  |     | Mysql  |     | Mysql  |   |
|   |   LB   +-----+   LB   +-----+   LB   +-----+   LB   |   |
|   +---+----+     +----+---+     +----+---+     +----+---+   |
|       |               |              |              |       |
|       +---------------+-------+------+--------------+       |
|       |                       |                     |       |
+-------------------------------------------------------------+
        |                       |                     |
        |                       |                     |
        |                       |                     |
+-------+--------+     +--------+-------+    +--------+-------+
|                |     |                |    |                |
|    MySQL01     |     |    MySQL02     |    |    MySQL03     |
|    Gluster01   +-----+    Gluster02   +----+    Gluster03   |
|                |     |                |    |                |
+----------------+     +----------------+    +----------------+

Then I mount them into the php containers like this:

--mount type=bind,source=/mnt/client-data,target=/var/www/html/

This works but this is very slow. The page loading times are around 10 seconds, when the files are not mounted (They exist in the container) the page loading times are around 2-3 seconds.

I came around Flocker and this seems very interesting, but I think a flocker volume can only be mounted on one container. Is this true?

Another solution that I am trying out right now, is that the code gets pulled from git every time a new container gets created. This is actually a good solution but it takes around 5 minutes to pull the code and run composer and when I push updates I need to restart all containers.

What would be the best solution to share my code amongst different containers on multiple hosts? (or even datacenters) Currently I have access to many different storage backends (Ceph, NFS, gluster) and creating a new one is no issue.

EDIT: Why is gluster unreliable in this case? I might have said it wrong above... Gluster is unreliable when using docker volumes on Gluster, with a volume driver plugin. The volumes get mounted nicely most of the time when creating a service, but in case rescheduling happens in the swarm the volumes rarely gets mounted again. I will look into this later to determine what goes wrong, but I don't have a lot of free time at the moment. Gluster is in fact, solid. Just not in this case.

Answer 1

What I eventually ended up doing is use CephFS and mount this on the docker hosts. The write speeds are OK, maybe even excellent when compared to glusterfs. The reads are extremely fast on CephFS, this was also the case with GlusterFS.

This setup suits my needs, but I can't use shared docker volumes accross hosts. I have to bind the mounted filesystem in docker containers. Although this is not the end of the world, I am still looking for a convenient and fast way to have my volumes accross all the hosts in the swarm.

Currently I am trying to build a Docker volume plugin for CephFS to achieve my requirements. I'll keep this updated in case someone is interested.

Answer 2

Note that I have not tried the following.

Since your code is mostly read-only, it seems to me that you "just" need a file system cache. CacheFS for example running on top of NFS.

Since GlusterFS can be mounted as NFS, you should be able to combine the two.