Save Prometheus Metrics

Question

I am trying to setup prometheus and grafana to save metrics from an express app Here is my docker-compose.yml file

version: '3'
services:
  prometheus:
    image: "prom/prometheus"
    network_mode: host
    ports:
      - "9090:9090"
    volumes:
      - /var/www/html/kubernetes/prometheus:/etc/prometheus
      - data:/prometheus-data
  grafana:
    image: "grafana/grafana"
    depends_on:
      - prometheus
    ports:
      - "1999:3000"
volumes:
  data:

Here is my express app code

const express = require('express')
const promBundle = require("express-prom-bundle");
const metricsMiddleware = promBundle({includeMethod: true,includePath:true, customLabels:{app: "service_a"}});
const app = express()
const port = 3000
app.use(metricsMiddleware);
app.get('/', (req, res) => {
   return res.json({
       message:"This is Service A"
   })
})
app.get('/login', (req, res) => {
    return res.status(400).json({
        message:"error"
    })
})
app.get('/sub', (req, res) => {
   return res.json({
       message:"This is Service A SubRoute"
   })
})
app.listen(port, () => {
    console.log(`Example app listening at http://localhost:${port}`)
})

This runs perfectly but the issue is that when i stop the express app the the metrics/logs in prometheus is empty, i was expecting to see old logs from prometheus. Probably saved in the mounted volume. What am i missing?

Answer 1

You are going to use /prometheus folder instead as volume. That should do.

Also not necessarily but depending on your host OS setup, you might need to chown to nobody the local directory - see this line .

Answer 2

You can keep all files related to Prometheus within your volume with:

volumes:
       - ...
       - data:/prometheus

Or you can keep only the data by saving the data folder inside the prometheus workdir, as per the documentation .