context deadline extended - prometheus - while running on docker

Question

I'm trying to get a spring microservice to regsiter on Prometheus. Both are running on docker. I get "context deadline exceeded" in the Prometheus UI for my service.

Funnily, when i try to open "localhost:8081/metrics" in a new tab, it shows the metrics.

This is my docker-compose.yml. This issue seems pretty common but I've not come across a solution that works for me ; yet.

docker-compose.yml

version: '2.1'
networks:
   cadrs:
     ipam:
       config:
          - subnet: 172.28.0.0/16

services:
prometheus:
    image: prom/prometheus:0.18.0
    volumes:
        - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
        - '-config.file=/etc/prometheus/prometheus.yml'
    ports:
        - '9090:9090'

demo:
    build: "C:/Users/hmt843/Downloads/demo"
    hostname: "demo"
    ports:
        - "8081:8081"
    networks:
       cadrs:
          ipv4_address: 172.28.1.2
node-exporter:
    image: prom/node-exporter:v0.15.0
    ports:
        - '9100:9100'

prometheus.yml

global:
  scrape_interval: 15s
  external_labels:
        monitor: 'my-monitor'

scrape_configs:
    - job_name: 'prometheus'
      target_groups:
          - targets: ['localhost:9090']
    - job_name: 'demo'
      tls_config:
        insecure_skip_verify: true
      target_groups:
          - targets: ['172.28.1.2:8081']
    - job_name: 'node-exporter'
      target_groups:
          - targets: ['node-exporter:9100']

Answer 1

By specifying the network section in your app's service the container is placed in a separate docker network called cadrs . Your Prometheus service has no such network definition and thus is in the default network. Containers must reside in at least one common network to be able to access each others. Even exposing the container on the host port doesn't affect this (but you could reach the app via your host ip - don't do this).

If you really need this network configuration, either add Prometheus to the same network by adding the same network section to your Prometheus service or introduce another network (eg monitoring ) that is used in both services. Alternatively you could also specify that your app is in the default network by adding one more line to your networks section of the app.

networks:
  cadrs:
    [...]
  default:

Reference: https://docs.docker.com/compose/networking/#configure-the-default-network

With this setup, you can modify your scrape configuration also back to demo:8081 . Either way, ask yourself if you really want to use manage container addresses by yourself...