504来自ngingx反向代理与Node.js + Docker的响应

Question

UPDATE

I have found out that it works if I restart nginx on the container with docker exec -it backend_gateway_1 nginx -s reload . I have found some information from this question . It seems that connecting to 127.0.0.1 rather than localhost would work, however I am using docker networks so I'm not sure what my situation would be.

ORIGINAL QUESTION

I am trying to use nginx as a reverse proxy with docker. I cannot quite work out what's wrong as sometimes it works when but then if I use docker-compose down && docker-compose up to restart it all, i will then usually get this 504 error

I have a feeling this is to do with the nginx or docker configuration. I am able to make a request to / fine but any request to /user is causing the intermittent issue.

When I am not receiving a 504, the node app is working well and creates/returns a user. I am also able to connect to the database fine with mongo-express so that should help narrow something down.

docker-compose.yml

version: '3.5'
services:
  gateway:
    image: nginx:latest
    restart: always
    ports:
      - 8080:80
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/nginx-error.log:/var/log/nginx/error.log
    networks:
      - poker_network
  poker_mongo:
    image: mongo:latest
    container_name: poker_mongo
    restart: unless-stopped
    volumes:
      - db-data:/data/db
    expose: 
      - "27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: test
      MONGO_INITDB_ROOT_PASSWORD: 123
    networks:
      - poker_network
    depends_on:
      - gateway
  poker_user:
    container_name: poker_user
    build: ./user/
    image: poker/user:latest
    volumes:
      - ./user/:/usr/src/app
      #- /usr/src/app/node_modules
    expose:
      - "8080"
    depends_on:
      - gateway
      - poker_mongo
    networks:
      - poker_network
    environment:
      WAIT_HOSTS: poker_mongo:27017
  mongo-express:
    container_name: mongo-express
    image: mongo-express
    restart: always
    ports:
      - 8085:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: test
      ME_CONFIG_MONGODB_ADMINPASSWORD: 123
      ME_CONFIG_MONGODB_SERVER: poker_mongo
    depends_on:
     - gateway
     - poker_mongo
    networks:
      - poker_network

networks:
  poker_network:
volumes: 
  db-data:

nginx.conf

events { worker_connections 1024; }

http {

    server {
        listen 80;
        location /user {
            rewrite /user/(.*) /$1  break;
            proxy_pass http://poker_user:8080/;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Host $host;
        }
        location / {
            return 200 'hello';
        }

    }
}

Any help is appreciated, thanks.

Answer 1

I found several similar related questions and will try to compile two possible solutions out of those:

• https://serverfault.com/questions/240476/how-to-force-nginx-to-resolve-dns-of-a-dynamic-hostname-everytime-when-doing-p
• Docker Network Nginx Resolver

First solution

First solution is to reverse dependency between services: instead of poker_user depend on gateway you can may gateway depend on user. This way when gateway starts if will be able to resolve poker_user hostname.

The Dockerfile would look this way in this case:

version: '3.5'
services:
  gateway:
    image: nginx:latest
    restart: always
    ports:
      - 8080:80
    depends_on: poker_user
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/nginx-error.log:/var/log/nginx/error.log
    networks:
      - poker_network
  poker_mongo:
    image: mongo:latest
    container_name: poker_mongo
    restart: unless-stopped
    volumes:
      - db-data:/data/db
    expose: 
      - "27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: test
      MONGO_INITDB_ROOT_PASSWORD: 123
    networks:
      - poker_network
  poker_user:
    container_name: poker_user
    build: ./user/
    image: poker/user:latest
    volumes:
      - ./user/:/usr/src/app
      #- /usr/src/app/node_modules
    expose:
      - "8080"
    depends_on:
      - poker_mongo
    networks:
      - poker_network
    environment:
      WAIT_HOSTS: poker_mongo:27017
  mongo-express:
    container_name: mongo-express
    image: mongo-express
    restart: always
    ports:
      - 8085:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: test
      ME_CONFIG_MONGODB_ADMINPASSWORD: 123
      ME_CONFIG_MONGODB_SERVER: poker_mongo
    depends_on:
     - gateway
    networks:
      - poker_network

networks:
  poker_network:
volumes: 
  db-data:

I also removed some other dependencies as I think they are redundant.

Second solution

This solution is based on changing nginx configuration. It looks like nginx tries to resolve hostname specified in proxy_pass upfront and since it's not available yet when nginx starts it will keep responding with 504 because it doesn't try to resolve it again.

There seems to be a workaround with using variables inside proxy_pass directive which are treated differently and force nginx to perform DNS resolution again. This requires resolver to be configured in nginx and luckily there's one built into Docker when using user-defined networks.

Here's an example nginx configuration (though I haven't tested it myself):

events { worker_connections 1024; }

http {

    server {
        listen 80;
        location /user {
            # Docker DNS server in user-defined networks
            # https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/
            resolver 127.0.0.11 ipv6=off;

            rewrite /user/(.*) /$1  break;

            # Using a variable will force nginx to re-resolve DNS name
            # https://serverfault.com/a/593003
            set $backend "http://poker_user:8080/";
            proxy_pass $backend;

            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Host $host;
        }
        location / {
            return 200 'hello';
        }

    }
}

Hope this help!