init docker swarm with docker machine: context deadline exceeded

Question

I'm learing Docker machine while encount some problems.
My computer is mac and use Docker for mac. I create 2 vm,vm1& vm2 by docker-machine,and try to init a swarm who has nodes-vm1,vm2 and my mac.My steps are below:
1. create an image called "sprinla/cms:latest" and a docker-compose.yml

version: "3"
services:
  web:
    image: sprinla/cms:latest
    deploy:
      replicas: 1
    ports:
      - "80:80"
    networks:
      - webnet
    command: /data/start.sh
networks:
  webnet:

2.create 2 vms.Here is vm info:

yuxrdeMBP:~ yuxr$ docker-machine ls  
NAME   ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS  
vm1    -        virtualbox   Running   tcp://192.168.99.100:2376           v17.12.0-ce  
vm2    -        virtualbox   Running   tcp://192.168.99.101:2376           v17.12.0-ce  

3. init swarm on my mac host:

yuxrdeMBP:~ yuxr$ docker swarm init
Swarm initialized: current node (uf6rg1v91exlwntlskyj8iim7) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-3qb32l84n0s8vl74rj9d6psm7bzdany3piw55ohtrq0q7ly814-c5km5zg3kj9d6vn6vrtt6xxtg 192.168.65.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

4 join vm1 to swarm,then comes the problem

yuxrdeMBP:~ yuxr$ docker-machine ssh vm1 "docker swarm join --token SWMTKN-1-3qb32l84n0s8vl74rj9d6psm7bzdany3piw55ohtrq0q7ly814-c5km5zg3kj9d6vn6vrtt6xxtg 192.168.65.2:2377"
Error response from daemon: Timeout was reached before node joined. The attempt to join the swarm will continue in the background. Use the "docker info" command to see the current swarm status of your node.
exit status 1

5.cat the docker log :

time="2018-01-03T17:13:50.387854642Z" level=debug msg="Calling GET /_ping"
time="2018-01-03T17:13:50.388228524Z" level=debug msg="Calling GET /_ping"
time="2018-01-03T17:13:50.388521374Z" level=debug msg="Calling POST /v1.35/swarm/join"
time="2018-01-03T17:13:50.388583426Z" level=debug msg="form data: {\"AdvertiseAddr\":\"\",\"Availability\":\"\",\"DataPathAddr\":\"\",\"JoinToken\":\"*****\",\"ListenAddr\":\"0.0.0.0:2377\",\"RemoteAddrs\":[\"192.168.65.2:2377\"]}"
time="2018-01-03T17:13:55.392578452Z" level=error msg="failed to retrieve remote root CA certificate" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=node
time="2018-01-03T17:14:02.394608777Z" level=error msg="failed to retrieve remote root CA certificate" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=node
time="2018-01-03T17:14:09.395720474Z" level=error msg="failed to retrieve remote root CA certificate" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=node
time="2018-01-03T17:14:10.393743738Z" level=error msg="Handler for POST /v1.35/swarm/join returned error: Timeout was reached before node joined. The attempt to join the swarm will continue in the background. Use the \"docker info\" command to see the current swarm status of your node."
time="2018-01-03T17:14:16.398095265Z" level=error msg="failed to retrieve remote root CA certificate" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=node
time="2018-01-03T17:14:23.399587783Z" level=error msg="failed to retrieve remote root CA certificate" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=node
time="2018-01-03T17:14:25.399943337Z" level=error msg="cluster exited with error: rpc error: code = DeadlineExceeded desc = context deadline exceeded"

6. below is my mac ifconfig info:

   yuxrdeMBP:~ yuxr$ ifconfig
   lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
     options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
     inet 127.0.0.1 netmask 0xff000000
     inet6 ::1 prefixlen 128
     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
     nd6 options=201<PERFORMNUD,DAD>
   gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
   stf0: flags=0<> mtu 1280
   XHC20: flags=0<> mtu 0
   en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
     ether ac:bc:32:81:97:37
     inet6 fe80::4d8:6b2:718a:5d3b%en0 prefixlen 64 secured scopeid 0x5
     inet 192.168.199.169 netmask 0xffffff00 broadcast 192.168.199.255
     nd6 options=201<PERFORMNUD,DAD>
     media: autoselect
     status: active
   p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
     ether 0e:bc:32:81:97:37
     media: autoselect
     status: inactive
   awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
     ether 36:9f:65:fd:34:c3
     inet6 fe80::349f:65ff:fefd:34c3%awdl0 prefixlen 64 scopeid 0x7
     nd6 options=201<PERFORMNUD,DAD>
     media: autoselect
     status: active
   en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
     options=60<TSO4,TSO6>
     ether 6a:00:00:e3:4c:30
     media: autoselect <full-duplex>
     status: inactive
   en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
     options=60<TSO4,TSO6>
     ether 6a:00:00:e3:4c:31
     media: autoselect <full-duplex>
     status: inactive
   bridge0: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500
     options=63<RXCSUM,TXCSUM,TSO4,TSO6>
     ether 6a:00:00:e3:4c:30
     Configuration:
         id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
         maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
         root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
         ipfilter disabled flags 0x2
     member: en1 flags=3<LEARNING,DISCOVER>
             ifmaxaddr 0 port 8 priority 0 path cost 0
     member: en2 flags=3<LEARNING,DISCOVER>
             ifmaxaddr 0 port 9 priority 0 path cost 0
     media: <unknown type>
     status: inactive
   utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
     options=6403<RXCSUM,TXCSUM,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
     inet6 fe80::441e:c0e3:5429:2abb%utun0 prefixlen 64 scopeid 0xb
     nd6 options=201<PERFORMNUD,DAD>
   utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
     options=6403<RXCSUM,TXCSUM,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
     inet6 fe80::7820:5bac:4735:7f82%utun1 prefixlen 64 scopeid 0xc
     inet6 fd44:5cb3:4ab4:5d08:7820:5bac:4735:7f82 prefixlen 64
     nd6 options=201<PERFORMNUD,DAD>
   utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
     options=6403<RXCSUM,TXCSUM,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
     inet6 fe80::26f2:e964:8dfb:e884%utun2 prefixlen 64 scopeid 0xd
     nd6 options=201<PERFORMNUD,DAD>
   gpd0: flags=8862<BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1400
     ether 02:50:41:00:01:01
   vboxnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
     ether 0a:00:27:00:00:00
     inet 192.168.99.1 netmask 0xffffff00 broadcast 192.168.99.255
   

Why????
mac host has ip, 192.168.99.1 ,vm1 has ip 192.168.99.100,vm2 has ip 192.168.99.101,they are in the same network,why can't vm1 nor vm2 join the mac host's swarm?

ANOTHER QUESTION:if i use vm1 as swarm manager,run "docker swarm join" commad on the mac host,when join as worker,it can join but can't use;when join as manager will has error:

yuxrdeMBP:~ yuxr$ docker swarm join --token SWMTKN-1-49w1hd28hs1mtj3sgmd0o3q7n59zgppvd18vs0iwhcnjemzmwb-7mk35zdnaslt1p41gninvwlud 192.168.99.100:2377
Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unknown desc = could not connect to prospective new cluster member using its advertised address: rpc error: code = Unavailable desc = grpc: the connection is unavailable

THANK YOU FOR HELP ME !!!

Answer 1

There is no routing between the Mac host and Docker for Mac. So on a Mac you can only setup multi-node swarms between VMs , and the standard Docker for Mac cannot participate in a multi-node swam . This is a limitation on how networking is implemented on OSX.

See the documentation , where this is explained. Also see this issue for more background.

Answer 2

For Me, this error got resolved by making the Security groups to Inbound Rules to All traffic in AWS.

Answer 3

I got the same error when trying to join a swarm cluster as a worker Used 2 VMs from Google cloud for this..

Manager node was working fine ..docker info--> swarm did not give any errors. but when i try to join the worker nodes with the token .. i got this error "Error response from daemon: Timeout was reached before node joined. The attempt to join the swarm will continue in the background. Use the "docker info" command to see the current swarm status of your node. " while docker info showed me "rpc error: code = DeadlineExceeded desc = context deadline exceeded in swarm error"

tried a lot of different things finally below solution worked.

solution. -->. i used "docker swarm init --force-new-cluster". in one of the vms i tried to join the as a worker.. and then i used "docker swarm leave --force" on the existing manager node .. and the joined that one as a worker to the newly created cluster. Other vm also also worked when tried to join as workers for the new cluster..

ubuntu - 18.04 docker version -20.10.17