docker reverse proxy : can't get jwilder/nginx reverse proxy work with serveral bridge networks

Question

I'm trying to have several website on a single VPS (based on prestashop) and am using docker to manage the whole thing. VPS is running DEBIAN 10.

I first have a jwilder/nginx-proxy container, here is the docker-compose.yml:

version: '3.8'

services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    container_name: c_reverse_proxy
    ports:
      - 80:80
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - net_reverse_proxy

networks:
 net_reverse_proxy:
  driver: bridge

I then want to dynamicaly create instances of prestashops with the following docker-compose.yml, created and execute by a bash script aa function of the subdomain passed as an argument:

docker-compose.yml for subdomain "testben":

version: '3.8'

services:
 db:
  image: mysql:5.7
  container_name: c_mysql_testben
  restart: always
  volumes:
   - /home/ben/shops/testben/mysql:/var/lib/mysql
  environment:
   MYSQL_ROOT_PASSWORD: AfeeRt0
  networks:
   - net_testben

 presta:
  image: prestashop/prestashop
  container_name: c_presta_testben
  expose:
   - 80
  restart: always
  volumes:
   - /home/ben/shops/testben/www:/var/www/html
   - /home/ben/shops/scripts/post_inst:/tmp/post-install-scripts
  environment:
   DB_SERVER: c_mysql_testben
   DB_USER: root
   DB_PASSWD: xxxxxxx
   PS_INSTALL_AUTO: 1
   PS_ERASE_DB: 0
   PS_LANGUAGE: fr
   PS_COUNTRY: FR
   PS_FOLDER_ADMIN: _admin
   PS_DOMAIN: testben.domain.com
   VIRTUAL_HOST: testben.domain.com
  depends_on:
   - db
  networks:
   - net_testben

networks:
 net_testben:
  driver: bridge

Then I connect the nginx reverse proxy container to the newly created bridge network:

docker network connect testben_net_testben c_reverse_proxy

At this stage, everything is connected, and I go to the web browser!

This doesn't work, and I get in firefox when trying to reach testben.domain.com

502 Bad Gateway nginx/1.17.6

Here are some debug info:

> docker logs c_reverse_proxy


dockergen.1 | 2020/08/07 16:01:04 Received event start for container 0cb8bc80f4fd
dockergen.1 | 2020/08/07 16:01:04 Generated '/etc/nginx/conf.d/default.conf' from 3 containers
2020/08/07 16:01:18 [notice] 50#50: signal process started
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
dhparam generation complete, reloading nginx
nginx.1    | 2020/08/07 16:01:26 [error] 51#51: *1 no live upstreams while connecting to upstream, client: XXX.XXX.XXX.XXX, server: testben.domain.com, request: "GET / HTTP/1.1", upstream: "http://testben.domain.com/", host: "testben.domain.com"
nginx.1    | testben.domain.com XXX.XXX.XXX.XXX - - [07/Aug/2020:16:01:26 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0"

reverse proxy automatic generated configuration:

    > docker exec c_reverse_proxy cat /etc/nginx/conf.d/default.conf
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
  default $http_x_forwarded_port;
  ''      $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header
map $scheme $proxy_x_forwarded_ssl {
  default off;
  https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
                ssl_protocols TLSv1.2 TLSv1.3;
                ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
                ssl_prefer_server_ciphers off;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        listen 80;
        access_log /var/log/nginx/access.log vhost;
        return 503;
}
# http://testben.domain.com
upstream http://testben.domain.com {
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
}
server {
        server_name http://testben.domain.com;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        location / {
                proxy_pass http://testben.domain.com;
        }
}

I think the issue is around:

 upstream http://testben.domain.com {
                                     # Cannot connect to network of this container
                                     server 127.0.0.1 down;
     }

It seems to be close to this open issue

Here is where I am:

• DNS is well configured
• prestashop docker-compose.yml is fine, if I add port: - 8080:80, I can reach te site without the proxy
• nginx container is well connected to the dedicated bridge network and can see the prestashop container starting

I would take any help, clue and advise. What I would prefer avoiding is having all containers on a single bridge network, as I would like to have maximal separation between every instance of prestashop I will run on this server.

Thanks for taking the time to read.

Answer 1

Ok I found the solution by my-self. Posting to help others!

The issue is I was connecting the separate network ( testben_net_testben ) to nginx reverse proxy container after the container to be proxied were created. So at prestashop container creation, the reverse proxy container reads on socket the container is created, but at this time, when generating the default.conf, it can't reach the prestashop container.

Solution is to create the network first, connect it to the reverse proxy, and then start the main service, with reference to external created network.

Step 1:

docker network create net_testben

Step 2:

docker network connect net_testben c_reverse_proxy

Step 3:

docker-compose up -d

with following docker-compose.yml

services:
 db:
  image: mysql:5.7
  container_name: c_mysql_testing
  restart: always
  volumes:
   - /home/ben/shops/testben/mysql:/var/lib/mysql
  environment:
   MYSQL_ROOT_PASSWORD: AfeeRt0
  networks:
   - net_testben

 presta:
  image: prestashop/prestashop
  container_name: c_presta_testing
  expose:
   - 80
  restart: always
  volumes:
   - /home/ben/shops/testben/www:/var/www/html
   - /home/ben/shops/scripts/post_inst:/tmp/post-install-scripts
  environment:
   DB_SERVER: c_mysql_testing
   DB_USER: root
   DB_PASSWD: xxxx
   PS_INSTALL_AUTO: 1
   PS_ERASE_DB: 0
   PS_LANGUAGE: fr
   PS_COUNTRY: FR
   PS_FOLDER_ADMIN: _admin
   PS_DOMAIN: testben.domain.com
   VIRTUAL_HOST: testben.domain.com
  depends_on:
   - db
  networks:
   - net_testben

networks:
 net_testben:
  external : true