如何使用 Prometheus Operator 抓取 Jenkins 指标

Question

I'm using Kube-prometheus with Prometheus-Operator to monitor my K8s cluster. I've deployed Jenkins on my cluster and want to start to get metrics here using ServiceMonitor.

I've installed the Prometheus plugin which exposes the metrics using /prometheus or by /metrics/API_KEY/metrics , this works fine if I create a new static job. However, if I want to use ServiceMonitor, it does not work.

apiVersion: monitoring.coreos.com/v1                                                                                                                                                                        
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s 
    port: http
    path: /metrics/y1H6G16T-DhqpHdW9XwHWnP9FWAXMMfy4XnXVnyoIOEV3-gPJZKN284OFUcVkPxL/metrics
  selector:
    matchLabels:
      jenkins: main

Answer 1

I don't know about ServiceMonitor, but I monitor my Jenkins instance without any problem, using annotations on Jenkins' service :

prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/prometheus"

Answer 2

I'm using kube-prometheus-stack v12.8.0 (formerly known as the prometheus-operator helm chart).

To make prometheus-operator detect external serviceMonitors (like the one provided by Jenkins), you'll have to configure two things:

1. configure it to scan other namespaces:

   serviceMonitorNamespaceSelector matchLabels: prometheus: please-scan-this-namespace-too

note: alternatively, you can leave it to {} so that all namespaces are scanned

2. configure it to also select the serviceMonitors detected in these other namespaces:

   serviceMonitorSelector: matchLabels: release: prometheus-operator

note: even though the documentation states that if you leave serviceMonitorSelector to {}, it will select all serviceMonitors, it does not seem to work.

And finally, you'd still need to add these labels to 1) the namespaces and 2) serviceMonitors that you want prometheus to adopt.