collect metrics from different pods in prometheus

Question

i want to collect metrics from a deployment (with multiple pods) from Kubernetes, and on of my metrics is the number of calls that my deployment received, my question is about Prometheus, how can i tell Prometheus to call all the pods that are part of the deployment and collect metrics from them? And what is the best practice to achieve this goal?

Answer 1

I would highly recommend using prometheus-operator to do all heavy lifting with configuring Prometheus monitoring for your applications.

For example, having the Deployment and Service like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: fabxc/instrumented_app
        ports:
        - name: web
          containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: web
    port: 8080

You may configure ServiceMonitor object which will use Service as a service discovery endpoint to find all the pods of the Deployment. This assumes that your application is exposing metrics using HTTP path /metrics .

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web

This will make Prometheus scrape metrics for your application.

You may read more about ServiceMonitors here: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md