curl from docker on cluster node to master node

Question

I have a cluster with one master node and 3 compute nodes. A docker container on nodes 1 and 2 are able to "curl" to the master node. However node-3 fails with "Connection refused"

• ping from container works
• ping and curl from container's host works (ie node-3 to master)
• shutting down docker0 worked for node-1 and 2. not for node-3 ( https://serverfault.com/questions/642981/docker-containers-cant-resolve-dns-on-ubuntu-14-04-desktop-host )
• did not help: setsebool -P httpd_can_network_connect on
• firewalld is inactive on other nodes, so tried shutting it down on node-3 as well

what else can I look for?

UPDATE

thanks to @sxm1972, found this possibly unwanted entry while listing "docker network ls"

f49b6c9541d2 docker_gwbridge bridge local

how do I get rid of this?

deleted it using: docker network rm f49b6c9541d2 sudo service docker restart

but that did not solve curl issue

UPDATE

Then looked at /etc/hosts node3 contains an unexpected line:

192.168.1.1     079543e73dc7

the other nodes instead have an IP like 172.17.0.x

192.168.1.1 refers to master node's IP.

UPDATE

ifconfig on node1 and node2 host show 172.17.0.x for docker0 interface

ifconfig on node3 shows 192.168.xx: this seems corrupt.

I might have inadvertently done something to node3 IP. How can I restore it?

UPDATE

Tried

ip link del docker0
sudo service docker restart

host docker0 IP still seems to be the wrong one

SOLUTION

Thanks @sxm1972! the following worked:

sudo service docker stop
create/edit daemon.json from the link below setting bip to 172.17.0.1/16
ip link del docker0
sudo service docker restart

https://success.docker.com/article/How_do_I_configure_the_default_bridge_(docker0)_network_for_Docker_Engine_to_a_different_subnet

I can now "curl" from a container

Answer 1

PS C:\Users\smallya> docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
857787541f21        bridge              bridge              local
f3de5219b77d        host                host                local
72ab4639a2dd        none                null                local
PS C:\Users\smallya> docker network inspect 857787541f21
[
    {
        "Name": "bridge",
        "Id": "857787541f210cb0912b3be240922810f007b149f200ae9fff5bdc111af14f14",
        "Created": "2018-02-09T14:31:17.0933179Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]