Nginx reverse proxy fails to access prometheus container

Question

I've created this docker compose file which build + runs without errors

version: '3.7'

volumes:
  grafana:
    driver: local
  prometheus-data:
    driver: local

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    expose:
      - 9090
    volumes:
      - prometheus-data:/prometheus
      - ./prometheus:/etc/prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
  
  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    expose:
      - 3000
    volumes:
      - grafana:/etc/grafana
    restart: unless-stopped
  
  nginx:
    image: nginx
    container_name: webserver
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /opt/certs/:/etc/nginx/certs/
      - ./nginx/gw-web/:/usr/share/nginx/html:ro
      - ./nginx/nginx_proxy.conf:/etc/nginx/conf.d/default.conf
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
    command: [nginx-debug, '-g', 'daemon off;']

nginx has the following reverse-proxy configuration:

server {
    listen 80;
    listen [::]:80;
    server_name 10.23.225.72;

    location /prometheus {

        proxy_pass http://prometheus:9090;

        proxy_set_header   Host             $host;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_max_temp_file_size 0;
        proxy_read_timeout 3000;    
    }


    location /grafana {

        proxy_pass http://grafana:3000;

        proxy_set_header   Host             $host;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_max_temp_file_size 0;
        proxy_read_timeout 3000;    
    }

    location / {
       root /usr/share/nginx/html;
    }
}

The root page is accessible and works as expected but when I try to go to any of the containers I just get a '404: not found' error.

What I've also tried:

    location /prometheus {

        proxy_pass http://prometheus:9090/;

and

    location /prometheus/ {

        proxy_pass http://prometheus:9090/;

and

    location /prometheus/ {

        proxy_pass http://prometheus:9090;

I've also tried running nginx as a service on the host machine, no success there either and anyway my project requires me to containerize this.

For shits and giggles, I tried adding another nginx container to the docker compose file:

  nginx2:
    image: nginx
    container_name: webserver2
    restart: unless-stopped
    expose:
      - 8080

and nginx config:

    location /webserver {

        proxy_pass http://webserver2:8080/;

        proxy_set_header   Host             $host;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_max_temp_file_size 0;
        proxy_read_timeout 3000;    
    }

But that also returns 404.

Is there something going on with the docker.networking maybe?

I check the.network configuration json by inspecting it from vscode

{
    "Name": "frp-6_default",
    "Id": "940872d1cf151f8d1bb060edbf95c1bcfe29a544c7ea9561091b79a6c5588510",
    "Created": "2023-01-25T11:52:01.520893117Z",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": null,
        "Config": [
            {
                "Subnet": "172.18.0.0/16",
                "Gateway": "172.18.0.1"
            }
        ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
        "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {
        "4a2c2b6ca8aec52f801ad554c6b2c14abbe616078393f1e8ef40bd82ad12aa2a": {
            "Name": "webserver2",
            "EndpointID": "86228e6678d82a9d6735b5440618be7fc281127e6d610d281d9ffc5bcb4f256f",
            "MacAddress": "02:42:ac:12:00:05",
            "IPv4Address": "172.18.0.5/16",
            "IPv6Address": ""
        },
        "9e13cbc4805fedf4db05b5de989f726bc110b97596b99b933a93e701641294a9": {
            "Name": "webserver",
            "EndpointID": "1bbcfe058373b225e5f53e8ff4d907d39bed578d9ac14d514ccf8da2d3c9628a",
            "MacAddress": "02:42:ac:12:00:04",
            "IPv4Address": "172.18.0.4/16",
            "IPv6Address": ""
        },
        "b846cc374fa09e90589c982613224f31135660a17a5f4585b24d876ea2abd53c": {
            "Name": "grafana",
            "EndpointID": "3c0dac6d612a48dc067a5c208334af4f90c8c1d9122e263ca464186c1cf19f35",
            "MacAddress": "02:42:ac:12:00:03",
            "IPv4Address": "172.18.0.3/16",
            "IPv6Address": ""
        },
        "dbe81d7b88413f78a06ce94f86781242299a3f6eee0f96f607ebaf8fb0b17be6": {
            "Name": "prometheus",
            "EndpointID": "4792bbd19cc7b782ec522d148eef7dcb5e31b5e38d99e886ff728a3f519b4973",
            "MacAddress": "02:42:ac:12:00:02",
            "IPv4Address": "172.18.0.2/16",
            "IPv6Address": ""
        }
    },
    "Options": {},
    "Labels": {
        "com.docker.compose.network": "default",
        "com.docker.compose.project": "frp-6",
        "com.docker.compose.version": "2.15.1"
    }
}

The following stand out to me:

    "Internal": false,
    "Attachable": false,

Can those be the culprits? If so, how do I change them? For the record, I also tried putting the IP addresses for the containers in the nginx config like so

    location /prometheus {

        proxy_pass http://172.18.0.2:9090/;

still unsuccessful...

Answer 1

The problem here is in your prometheus configuration. Because you have:

    location /prometheus {
        proxy_pass http://prometheus:9090;
        ...
    }

When you access http://yourhost/prometheus/ , that request gets proxied to http://prometheus:9090/prometheus/ , and by default prometheus doesn't know what to do with that /prometheus path.

You need to tell it that it's being served from a non-root path using the --web.external-url command line option. That might look something like:

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.console.libraries=/usr/share/prometheus/console_libraries
      - --web.console.templates=/usr/share/prometheus/consoles
      - --web.external-url=http://localhost:8080/prometheus/

(I've preserved all the command line options that are used by default in the prometheus image; the only thing new thing here is the --web.external-url option.)

---

Before making this change:

$ curl -i http://localhost:8080/prometheus
HTTP/1.1 404 Not Found
Server: nginx/1.23.3
Date: Wed, 25 Jan 2023 13:17:42 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 19
Connection: keep-alive
X-Content-Type-Options: nosniff

404 page not found

After making this change:

$ curl -i http://localhost:8080/prometheus
HTTP/1.1 302 Found
Server: nginx/1.23.3
Date: Wed, 25 Jan 2023 13:18:19 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 40
Connection: keep-alive
Location: /prometheus/graph

<a href="/prometheus/graph">Found</a>.