Docker Swarm Overlay - Encrypted Network and Firewall

Question

I set up a swarm on Digital Ocean using 5 Ubuntu 16.04 servers. They all are using the same Docker Engine version and firewall rules. It's set up to using 3 managers.

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 08:01:32 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 08:01:32 2017
 OS/Arch:      linux/amd64
 Experimental: false

22                         ALLOW       Anywhere                  
Anywhere/esp on eth1       ALLOW       Anywhere/esp               
2377/tcp on eth1           ALLOW       Anywhere                  
7946 on eth1               ALLOW       Anywhere                  
4789 on eth1               ALLOW       Anywhere                  
2376/tcp                   ALLOW       Anywhere                  
22 (v6)                    ALLOW       Anywhere (v6)             
Anywhere/esp (v6) on eth1  ALLOW       Anywhere/esp (v6)          
2377/tcp (v6) on eth1      ALLOW       Anywhere (v6)              
7946 (v6) on eth1          ALLOW       Anywhere (v6)              
4789 (v6) on eth1          ALLOW       Anywhere (v6)             
2376/tcp (v6)              ALLOW       Anywhere (v6)             

 app01     Ready   Active        
 app04     Ready   Active        Reachable
 app03     Ready   Active        Reachable
 app06     Ready   Active        Leader
 app05     Ready   Active    

I am using an encrypted network for my services.

[
    {
        "Name": "jupiter",
        "Id": "en4gdr54yw4w2xmo34fttdbvg",
        "Created": "0001-01-01T00:00:00Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097",
            "encrypted": "true"
        },
        "Labels": null
    }
]

I created my swarm to reside on eth1 . For example: docker swarm init --listen-addr eth1:2377 --advertise-addr eth1:2377 .

Despite this set up my 3 services can't see each other. As if the DNS doesn't work inside of the swarm. I found this link on github which describes what is happening to me.

One of my services tries to connect to the mongo service using the mongo servicename:port. But since the service connecting can't reach mongo it fails. Now I tried to trouble shoot the container by connecting inside of them using https://hub.docker.com/r/nicolaka/netshoot/ and it showed the containers could interact. (But I don't think it was connecting through the encryption set up for it.)

I think it must be the firewall but I am not an expert on networking security so I am wondering if I am missing something. I tried to compare what I had to what was written in issue 26523 concerning their firewall.

Anyhow I decided to create another network that wasn't encrypted and place my services on it.

Now the mongo is receiving connections:

2017-07-17T14:17:53.267+0000 I NETWORK  [conn8] received client metadata from 10.0.1.7:33424 conn8: { driver: { name: "nodejs", version: "2.2.29" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.4.0-62-generic" }, platform: "Node.js v6.11.1, LE, mongodb-core: 2.1.13" }
2017-07-17T14:17:53.369+0000 I NETWORK  [thread1] connection accepted from 10.0.1.7:33426 #9 (3 connections now open)
2017-07-17T14:17:53.373+0000 I NETWORK  [conn9] received client metadata from 10.0.1.7:33426 conn9: { driver: { name: "nodejs", version: "2.2.29" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.4.0-62-generic" }, platform: 

I would like to figure out how to get the encrypted network working. Is it the firewall, it seems to me it is, or is the Ubuntu server missing a package? ;<

Answer 1

After some time of being away from this I decided to revisit this again. Sometimes getting away helps.

I got it working now. I decided to add an additional rule with esp.

ufw allow to <local host eth1 ip> proto esp

ufw status

Anywhere/esp on eth1       ALLOW       Anywhere/esp
2377/tcp on eth1           ALLOW       Anywhere
7946 on eth1               ALLOW       Anywhere
4789 on eth1               ALLOW       Anywhere
2375/tcp on eth1           ALLOW       Anywhere
10.xx.xx.xx/esp            ALLOW       Anywhere
2376/tcp on eth1           ALLOW       Anywhere
22 (v6)                    ALLOW       Anywhere (v6)
Anywhere/esp (v6) on eth1  ALLOW       Anywhere/esp (v6)
2377/tcp (v6) on eth1      ALLOW       Anywhere (v6)
7946 (v6) on eth1          ALLOW       Anywhere (v6)
4789 (v6) on eth1          ALLOW       Anywhere (v6)
2375/tcp (v6) on eth1      ALLOW       Anywhere (v6)
2376/tcp (v6) on eth1      ALLOW       Anywhere (v6)