如何在 Prometheus/Grafana 中配置 Rabbitmq Metric Kubernetes

Question

My question i want to add rabbitmq monitoring in prometheus. I already have rabbitmq running in Kubernetes but i dont know how to add rabbitmq metric in prometheus

I have install promethues and grafana through yaml file along with pv,pvc,storage,svc,config,deploy and cluster-role

Here is the screenshot of rabbitmq showing empty in promethues

I have install Kubernetes in one vm with local storage ie., control-plane and node both install in one vm and everything is working fine

Here is my prometheus-config yaml file

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - "alertmanager.monitoring.svc:9093"

    scrape_configs:
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-nodes'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics

      
      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
      
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      
      - job_name: 'kubernetes-service-endpoints'

        kubernetes_sd_configs:
        - role: endpoints

        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name
      
      - job_name: 'rabbitmq'
        metrics_path: /metrics
        scrape_interval: 5s
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
              - default

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_label_app]
            separator: ;
            regex: rabbitmq
            replacement: $1
            action: keep
          - source_labels: [__meta_kubernetes_endpoint_port_name]
            separator: ;
            regex: prometheus

Even after adding rabbitmq metric it is not showing in prometheus url (target)

here is my rabbitmq-svc yaml file

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: xxx-rabbitmq
spec:
  selector:
    matchLabels:
      app: xxx-rabbitmq
  serviceName: xxx-rabbitmq
  replicas: 1
  template:
    metadata:
      labels:
        app: xxx-rabbitmq
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: xxx-rabbitmq
        image: rabbitmq:3.7.3-management
        ports:
        - containerPort: xxx
        - containerPort: xxx
        - containerPort: xxx
        - containerPort: xxx        
        volumeMounts:
        - name: xxx-rabbitmq-pvc
          mountPath: /var/lib/rabbitmq
          subPath: rabbitmq
        envFrom:
        - configMapRef:
            name: rabbitmq-config

  volumeClaimTemplates:
    - metadata:
        name: xxx-rabbitmq-pvc
      spec:
        accessModes: ["ReadWriteOnce"]
        volumeMode: Filesystem
        storageClassName: xxxx-storage
        resources:
          requests:
           storage: 10Gi  
---
apiVersion: v1
kind: Service
metadata:
  name: xxx-rabbitmq
  labels:
    app: xxx-rabbitmq
spec:
  type: ClusterIP
  ports: 
  - port: xxx
    name: main
  - port: xxx
    name: rabbitmqssl
  - port: xxx
    name: rabvitmqmgmt
  selector:
    app: xxx-rabbitmq

Please help me out how to get all information of rabbitmq in promethues/grafana

Answer 1

If you use rabbitmq of version 3.8 and above you have to use rabbitmq_prometheus plugin.

As the docs state:

The plugin exposes all RabbitMQ metrics on a dedicated TCP port, in Prometheus text format.

To enable it run:

rabbitmq-plugins enable rabbitmq_prometheus

To validate it's working run inside rabbitmq container:

curl -s localhost:15692/metrics | head -n 3

and see if you get metrics in response.

For versions prior to 3.8, you'd have to use prometheus_rabbitmq_exporter