Prometheus - Django : connection refused

Question

(This is my first time using Prometheus and I'm not very good with Docker/Django yet)

I'm running a Django project in a docker container, and Prometheus with docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

In my docker-compose.yml I have:

...
nginx-proxy:
        build:
            context: ./dockerfiles/nginx-proxy/
            args:
                - DOMAIN_NAME=local.my.url
        ports:
            - "80:80"
        depends_on:
            - api
            - ...
        volumes:
            - ./volumes/nginx-front/log/:/var/log/nginx

api:
        build:
            context: ./dockerfiles/api/
            args:
                - GUNICORN_WORKERS=20
        restart: always
        volumes:
            - ./volumes/api/src/:/usr/src/app
...

In /tmp/prometheus.yml I have:

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'my-project-monitor'

rule_files:

scrape_configs:
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'api'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['api.local.my.url']

The prometheus job seems to work ok (but those aren't the metrics I'm interested in), the api gives the following error from the Promotheus UI: Get http://api.local.my.url:80/metrics: dial tcp 127.0.0.1:80: connect: connection refused

However, when I type in http://api.local.my.url:80/metrics in my browser I can see the information correctly. I've tried replacing the URL with my IP address 10.25.2.192 but that doesn't change the result.

I don't understand why it can't connect.

Answer 1

It's because your prometheus container is on a different network and for it 'localhost' means a different thing. It purposefully does not have access to the host's network (by default).

You can verify that by running sudo docker network ls while both of your containers are running.

What you could do is make sure the two containers run on the same network.

In your docker command, that would mean adding --network [name] and for docker-compose, it would mean adding a network: attribute. That can be done for all the services in the docker-compose file if you add something like this at the bottom of your file:

networks:
  default:
    external:
      name: [name]

Source: https://docs.docker.com/compose/networking/#use-a-pre-existing-network

To actually create the network outside of docker-compose you can use

sudo docker network create -d bridge [name]