Multi-host Docker compose application on Swarm

Question

I've a simple compose file, where I've three services A, B and C. Both A and B depends on C (ie have links to C). Here a simplified excerpt from docker-compose.yml :

kafka:
  image: spotify/kafka
  environment:
    ADVERTISED_PORT: 9092
  ports:
    - "2181:2181"
    - "9092:9092"
ServiceA:
  image: elsoufy/myimage
  command: ./mycommand -role producer -queue kafka:9092
  ports:
    - "8080"
ServiceB:
  image: elsoufy/myimage
  command: ./mycommand -role consumer -queue kafka:9092

I've setup a Docker swarm on AWS and enabled overlay networking through consul key-store. I've been struggling for a while to get this working properly (I had to manually upgrade the kernel of the machines to linux 3.16 ).

I'm using Docker 1.9

Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      linux/amd64

I can successfully launch the compose up with docker-compose up -d , but ServiceA and ServiceB crashes after many attempt to connect to kafka as they cannot find it. When I try to use explicit links and enable overlay for this app I get an error :

docker-compose --x-networking --x-network-driver=overlay up

links, which are not compatible with Docker networking and will be ignored. Future versions of Docker will not support links - you should remove them for forwards-compatibility.

It looks like I cannot use links when I want to enable multi-host!! But then how I can run a compose application where I've dependencies between containers and scale the containers on a docker swarm?

Answer 1

You don't need explicit links anymore. Network should link all containers on the same user defined network. Compose creates a default network for you , based on the project name.

Answer 2

To express dependency between containers or services you can use the configuration option depends_on in the docker-compose.yml file, but version 2 file format.

So your docker-compose.yml file should look similar like this:

version: '2'
services:
    kafka:
      image: spotify/kafka
      environment:
        ADVERTISED_PORT: 9092
      ports:
        - "2181:2181"
        - "9092:9092"

    ServiceA:
      image: elsoufy/myimage
      command: ./mycommand -role producer -queue kafka:9092
      ports:
        - "8080"
      depends_on:
        - kafka    

    ServiceB:
      image: elsoufy/myimage
      command: ./mycommand -role consumer -queue kafka:9092
      depends_on:
        - kafka     

networks:
      ...

docker-compose up will start services in dependency order. In the above example, kafka will be started before ServiceA and ServiceB .

NOTE: if you're using version 1 file format along with links , your app will work, but Swarm will schedule all containers on one host, because links between containers do not work across hosts with the old networking system.