What are the possibilities to debug a docker swarm?

Question

I'm trying to create a container's cluster using the docker swarm engine and following the official documentation: https://docs.docker.com/swarm/install-w-machine/ . But unfortunately till now without success.

On the documentation's step to get the information about the swarm, only two containers are displayed (the ones from the swarm-master) instead of four. Here are the output I'm getting:

$ docker info
Containers: 2
Images: 7
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 11
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.5-boot2docker
Operating System: Boot2Docker 1.7.0 (TCL 6.3); master : 7960f90 - Thu Jun 18     18:31:45 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: swarm-master
ID: M4Z4:6ZBE:LV53:NHS3:7CT6:HNWM:GLQ4:5WPY:3CZR:HVAZ:GPP2:G7I3
Debug mode (server): true
File Descriptors: 20
Goroutines: 33
System Time: 2015-06-29T15:36:40.543771252Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

The current docker machines are:

$ docker-machine ls
local                     virtualbox   Running   tcp://192.168.99.100:2376
swarm-agent-00            virtualbox   Running   tcp://192.168.99.102:2376       swarm-master
swarm-agent-01            virtualbox   Running   tcp://192.168.99.103:2376       swarm-master
swarm-master     *        virtualbox   Running   tcp://192.168.99.101:2376       swarm-master (master)

are there possibilities to debug the swarm machines to find eventual wrong configurations ?

Answer 1

I found out that the there is a docker swarm instance running within the docker host. The connection I established was to the docker host and not to the swarm instance within it. The flag "--swarm" is the key to reach the swarm instance, and that was missing in my case.

Here is the command to check the environment variables of the docker host:

 $ docker-machine env swarm-master
 export DOCKER_TLS_VERIFY="1"
 export DOCKER_HOST="tcp://192.168.99.100:2376"
 export DOCKER_CERT_PATH="/home/bksoft/.docker/machine/machines/swarm-master"
 export DOCKER_MACHINE_NAME="swarm-master"
 # Run this command to configure your shell:
 # eval "$(docker-machine env swarm-master)"

And the command to check the docker swarm instance:

 $ docker-machine env --swarm swarm-master
 export DOCKER_TLS_VERIFY="1"
 export DOCKER_HOST="tcp://192.168.99.100:3376"
 export DOCKER_CERT_PATH="/home/bksoft/.docker/machine/machines/swarm-master"
 export DOCKER_MACHINE_NAME="swarm-master"
 # Run this command to configure your shell:
 # eval "$(docker-machine env swarm-master)"

Man recognize the different exposed ports "2376" and "3376".

Now the output of the docker info shows what I expected:

$ docker info
Containers: 5
Images: 3
Role: primary
Strategy: spread
Filters: affinity, health, constraint, port, dependency
Nodes: 3
 swarm-agent-00: 192.168.99.101:2376
  â Containers: 1
  â Reserved CPUs: 0 / 1
  â Reserved Memory: 0 B / 1.022 GiB
  â Labels: executiondriver=native-0.2, kernelversion=4.0.5-boot2docker,     operatingsystem=Boot2Docker 1.7.0 (TCL 6.3); master : 7960f90 - Thu Jun 18     18:31:45 UTC 2015, provider=virtualbox, storagedriver=aufs
 swarm-agent-01: 192.168.99.102:2376
  â Containers: 1
  â Reserved CPUs: 0 / 1
  â Reserved Memory: 0 B / 1.022 GiB
  â Labels: executiondriver=native-0.2, kernelversion=4.0.5-boot2docker,     operatingsystem=Boot2Docker 1.7.0 (TCL 6.3); master : 7960f90 - Thu Jun 18     18:31:45 UTC 2015, provider=virtualbox, storagedriver=aufs
 swarm-master: 192.168.99.100:2376
  â Containers: 3
  â Reserved CPUs: 0 / 1
  â Reserved Memory: 0 B / 1.022 GiB
  â Labels: executiondriver=native-0.2, kernelversion=4.0.5-boot2docker,     operatingsystem=Boot2Docker 1.7.0 (TCL 6.3); master : 7960f90 - Thu Jun 18     18:31:45 UTC 2015, provider=virtualbox, storagedriver=aufs
CPUs: 3
Total Memory: 3.065 GiB

Also the output of the virtual machines is different. No asterisk is displayed to point the current active machine:

$ docker-machine ls
NAME             ACTIVE   DRIVER       STATE     URL                             SWARM
swarm-agent-00            virtualbox   Running   tcp://192.168.99.101:2376       swarm-master
swarm-agent-01            virtualbox   Running   tcp://192.168.99.102:2376       swarm-master
swarm-master              virtualbox   Running   tcp://192.168.99.100:2376       swarm-master (master)

Hope this may help someone else.