Docker-Compose dynamically assigned ports with Consul

Question

I'm running a number of Spring Boot applications (with Actuator) that use Consul for service discovery. If I start the applications using docker-compose, specifying both the Host and Container port for each of the containers then they register correctly with Consul and pretty quickly get marked as healthy.

spring:
  application:
    name: myapp-A
  profiles:
    active: default 
  cloud:
    consul:
      enabled: true
      host: consul
      port: 8500
      discovery:
        prefer-ip-address: true
        healthCheckUrl: http://docker-host:${server.port}/health

The host address values for "consul" and "docker-host" are passed in as environment variables in the docker-compose yml file on startup and in simplistic terms relate to the ip address of the docker host machine (I am running Consul on the same server).

Please also note the use of the :

${server.port}

variable to tell Consul how to connect back to the container to check its health (it just uses Actuator in this respect). This needs to be an externally accessible address and port so that Consul can check the services.

This all works fine when I specify both the Host and Container port numbers in the docker-compose file for the containers.

I have got to the point where I want to scale up a few of the containers, but when I try to execute the

docker-compose up -d --scale myapp-A=5

command it tells me that it's not possible as the Host port has been specified. If I change the docker-compose yml file to only define the container port, and leave the Host port to be dynamically assigned it all then goes wrong as Consul (running externally) gets given the wrong URL to connect to and marks the containers as being unhealthy.

The question I have is how do I tell Consul the dynamically and "externally" exposed port number to use from my app running inside the container?

Is there some way to get the port number and pass that into the container at the point of creation?

I'd like to try and crack this using Compose if I can (rather than using Swarm, or by running Consul in another container in the same composed set of containers).

Answer 1

Scaling on a single host and port mapping don't often go together very well.

If you want to support dynamically mapped ports then you might need service discovery for your service discovery. As the spring apps themselves will never know the external mapped port you need to feed the information from the Docker daemon somehow.

On the Docker side, as you are already using consul, Registrator can publish container information as services:

→ docker ps -f name=verdaccio
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                    NAMES
a9f0726027fd        deployable/verdaccio:2.7.3   "node --trace_gc /ap…"   3 days ago          Up 12 hours         0.0.0.0:4873->4873/tcp   verdaccio

Which ends up in consul as a service:

→ curl -s 10.8.8.8:8521/v1/catalog/service/verdaccio | jq
[
  {
    "ID": "fe50cd37-576f-fbce-cb63-567359c87257",
    "Node": "c8fecf7ed36a",
    "Address": "172.16.231.22",
    "Datacenter": "dc1",
    "TaggedAddresses": {
      "lan": "172.16.231.22",
      "wan": "172.16.231.22"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceID": "registrator:verdaccio:4873",
    "ServiceName": "verdaccio",
    "ServiceTags": [
      "mhmb"
    ],
    "ServiceAddress": "",
    "ServicePort": 4873,
    "ServiceEnableTagOverride": false,
    "CreateIndex": 139,
    "ModifyIndex": 139
  }
]

Ports are available in SRV DNS records:

→ dig @127.0.0.1 -p 8621 verdaccio.service.consul. SRV

; <<>> DiG 9.8.3-P1 <<>> @127.0.0.1 -p 8621 verdaccio.service.consul. SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4713
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;verdaccio.service.consul.  IN  SRV

;; ANSWER SECTION:
verdaccio.service.consul. 0 IN  SRV 1 1 4873 c8fecf7ed36a.node.dc1.consul.

;; ADDITIONAL SECTION:
c8fecf7ed36a.node.dc1.consul. 0 IN  A   172.16.231.22
c8fecf7ed36a.node.dc1.consul. 0 IN  TXT "consul-network-segment="

;; Query time: 2 msec
;; SERVER: 127.0.0.1#8621(127.0.0.1)
;; WHEN: Thu Mar  1 23:08:15 2018
;; MSG SIZE  rcvd: 142

From there though, I'm not sure how you can configure Spring Cloud Consul to use this information. Does Spring Cloud include some feature to inject dynamic properties? If you could get Spring to look up the SRV record or make the API request before publishing the service then you are sorted. Alternatively Consul would need to support health checking SRV records directly which I don't believe it does.