rolling deployment for docker containers behind load balancer

Question

I have a problem with rolling deployments of docker containers behind a load balancer.

Here is my docker compose yml file contents.

nginx:
    image: nginx_image
    links:
        - node1:node1
        - node2:node2
        - node3:node3
    ports:
        - "80:80"
node1:
    image: nodeapi_image
    ports:
        - "8001"
node2:
    image: nodeapi_image
    ports:
        - "8001"
node3:
    image: nodeapi_image
    ports:
        - "8001"

and here my nginx.conf

worker_processes 4;

events { worker_connections 1024; }

http {

  upstream node-app {
        least_conn;
        server node1:8001 weight=10 max_fails=3 fail_timeout=30s;
        server node2:8001 weight=10 max_fails=3 fail_timeout=30s;
        server node3:8001 weight=10 max_fails=3 fail_timeout=30s;
  }

  server {
        listen 80;
        listen 443 ssl;

        # ssl    on;
        ssl_certificate     /etc/nginx/ssl/imago.io.chain.crt;
        ssl_certificate_key /etc/nginx/ssl/imago.io.key;

        location / {
          proxy_pass http://node-app;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection 'upgrade';
          proxy_set_header Host $host;
          proxy_cache_bypass $http_upgrade;
        }
  }
}

If I have a new built image I want to deploy I have to stop a node container, remove it and recreate it with the new image. The problem here is that the new container will get a new IP and the nginx container doesnt know about that new IP, so if I recreate the 3 containers behind the load balancer once I recreate the last one the app wont serve any more because all IPs in the nginx machines /etc/hosts and environment vairables are not up to date any more.

I could SSH in to each container, update its code by pulling from the git repo and restart the process but that seems just wrong to me. What is the right way to do this?

Answer 1

There is an easier way to achieve this, take the following docker-compose.yml file as an example :

lb:
    image: tutum/haproxy
    links:
        - app
    ports:
        - "80:80"
app:
    image: tutum/hello-world

This docker compose file describes two services :

• lb: a load balancer which uses the tutum/haproxy image
• app: a sample webapp listening on port 80

If you start those services naïvely with docker-compose up -d , you will end up with only 2 containers (the load balancer and the web app).

But if you run docker-compose scale app=3 then run again docker-compose up -d , you will end up with 4 load-balanced containers.

The key player here is the tutum/haproxy docker image which is able to discover the different containers it is linked to.

---

A similar solution is to use Jason Wilder's nginx-proxy image which has the advantage of discovering the new nodes live ; so you won't have to restart the lb service.

lb:
    image: jwilder/nginx-proxy
    volumes:
        - /var/run/docker.sock:/tmp/docker.sock:ro
    ports:
        - "80:80"
app:
    image: tutum/hello-world
    environment:
        VIRTUAL_HOST: www.mysite.com

The VIRTUAL_HOST environment variable must be set to the domain name that resolves to the IP address of your docker host.

---

Another one is to use Traefik

lb:
  image: traefik
  command: --docker
  ports:
    - "80:80"
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

app:
  image: tutum/hello-world
  labels:
    traefik.frontend.rule: Host:www.mysite.com

The traefik.frontend.rule label must define a Traefik rule set to the domain name that resolves to the IP address of your docker host.

Traefik also offers different load balancing strategies and circuit breakers .