Docker Swarm Access Container to Container on Published Port?

Question

I've seen at least 10 questions like this, but couldn't find an answer.

I'm moving from Rancher to Docker Swarm.

I used docker-compose to deploy a stack.

I'm using linux hosts.

➜  swarm git:(master) ✗ docker --version
Docker version 20.10.9, build c2ea9bc

➜  swarm git:(master) ✗ docker-compose --version
docker-compose version 1.29.2, build 5becea4c

The issue I'm having is I have a container (service) smtp that listens on port 2525 . Inside the swarm from other services I need to reference this service as smtp:25 not 2525 . Naturally I just did a port mapping. But the other services cannot access the service on port 25 only on 2525 .

Note that for testing I'm trying the mapping as 2526->2525 to rule out any privilege issues.

My docker-compose entry looks like:

smtp:
    image: <my repository>
    user: node
    environment:
      - NODE_ENV=production
    privileged: true
    ports:
      - published: 2526
        target: 2525
        protocol: tcp
        mode: ingress # I don't need this on the host, just on the swarm vip.
    deploy:
      replicas: 2
      placement:
        constraints:
          - node.labels.scheme == task
      restart_policy:
        condition: on-failure
        delay: 5s # how long to wait between retarts
        window: 20s # How long to determine if startup has succeeded
    healthcheck:
      test: netstat -l | grep 2525
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 20s

From a busybox container, I can successfully telnet smtp 2525 but telnet smtp 2526 gives me connection refused . Also, I cannot telnet to smtp:2526 from inside the smtp container either.

Docker inspect on the service reveals

 "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 2525,
                        "PublishedPort": 2526,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 2525,
                    "PublishedPort": 2526,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "wv8fe0yih5ql9ye09chcl6y1b",
                    "Addr": "10.0.0.41/24"
                },
                {
                    "NetworkID": "5a9tjqidsvphi9lphldit62tm",
                    "Addr": "10.0.2.145/24"
                }
            ]
        }

Netstat reveals

~/app $ netstat -tulpn | grep LISTEN
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:2525            0.0.0.0:*               LISTEN      13/node             
tcp        0      0 127.0.0.11:42381        0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:783           0.0.0.0:*               LISTEN      -        

Any ideas? Am I not allowed to do this?

Answer 1

First, on docker swarm, you use docker stack deploy not docker compose to deploy services.

Docker swarm manages service connections with overlay networks: When you deploy a stack ( docker stack deploy ) an overlay network is implicitly created and the services are attached to it. All services attached to an overlay network can discover each other using docker's mesh networking dns discovery.

So, in practical terms, you could deploy your smtp service with a stack file modified to look more like this:

version: "3.9"

networks:
  public:
    attachable: true
    driver: overlay
    name: smtp

services:
  smtp:
    image: ...
    command: .... --listen :25
    networks:
      - public
    ports:
      - 2525:25
    healthcheck:
      test: netstat -l | grep 25
    ...

Further caveats: there is no port remapping when services connect to each other using dockers bridge or overlay networks. The "ports" directive only applies to ingress, that is, packets arriving on the docker nodes from outside the swarm, that need to be routed to services. So, for service-to-service traffic to communicate over :25, then the smtp container needs to be configured to listen on :25.

Once that is done you can connect to smtp from containers and services by specifying the target network:

So, I assume the smtp service can be configured to listen on :25 I attach the service explicitly to a network called "public" in the compose file. This network is created with the global name "smtp", and it is marked attachable so ad-hoc containers can be attached to it. Its explicitly declared as "overlay" as docker compose would deploy this compose and try and create it as a bridge network.

Then, to verify that this works, the netshoot container can be used. Simply attach it to the now, available network and verify that dns resolves the "smtp" name on that network, and port 25 is reachable.

docker run --network smtp --rm -it nicolaka/netshoot
$ telnet smtp 25