如何在 Prometheus/Grafana 中配置 Rabbitmq Metric Kubernetes

Question

我的问题我想在普罗米修斯中添加 rabbitmq 监控。 我已经在 Kubernetes 中运行了 rabbitmq 但我不知道如何在普罗米修斯中添加 rabbitmq 指标

我已经通过 yaml 文件以及 pv、pvc、storage、svc、config、deploy 和 cluster-role 安装了 promethues 和 grafana

这是 rabbitmq 在 promethues 中显示为空的屏幕截图

我已经在一个带有本地存储的虚拟机中安装了 Kubernetes，即控制平面和节点都安装在一个虚拟机中，一切正常

这是我的 prometheus-config yaml 文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - "alertmanager.monitoring.svc:9093"

    scrape_configs:
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-nodes'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics

      
      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
      
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      
      - job_name: 'kubernetes-service-endpoints'

        kubernetes_sd_configs:
        - role: endpoints

        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name
      
      - job_name: 'rabbitmq'
        metrics_path: /metrics
        scrape_interval: 5s
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
              - default

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_label_app]
            separator: ;
            regex: rabbitmq
            replacement: $1
            action: keep
          - source_labels: [__meta_kubernetes_endpoint_port_name]
            separator: ;
            regex: prometheus

即使在添加 rabbitmq 指标后，它也没有在普罗米修斯 url （目标）中显示

这是我的 rabbitmq-svc yaml 文件

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: xxx-rabbitmq
spec:
  selector:
    matchLabels:
      app: xxx-rabbitmq
  serviceName: xxx-rabbitmq
  replicas: 1
  template:
    metadata:
      labels:
        app: xxx-rabbitmq
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: xxx-rabbitmq
        image: rabbitmq:3.7.3-management
        ports:
        - containerPort: xxx
        - containerPort: xxx
        - containerPort: xxx
        - containerPort: xxx        
        volumeMounts:
        - name: xxx-rabbitmq-pvc
          mountPath: /var/lib/rabbitmq
          subPath: rabbitmq
        envFrom:
        - configMapRef:
            name: rabbitmq-config

  volumeClaimTemplates:
    - metadata:
        name: xxx-rabbitmq-pvc
      spec:
        accessModes: ["ReadWriteOnce"]
        volumeMode: Filesystem
        storageClassName: xxxx-storage
        resources:
          requests:
           storage: 10Gi  
---
apiVersion: v1
kind: Service
metadata:
  name: xxx-rabbitmq
  labels:
    app: xxx-rabbitmq
spec:
  type: ClusterIP
  ports: 
  - port: xxx
    name: main
  - port: xxx
    name: rabbitmqssl
  - port: xxx
    name: rabvitmqmgmt
  selector:
    app: xxx-rabbitmq

请帮助我如何在 promethues/grafana 中获取 rabbitmq 的所有信息

Answer 1

如果你使用3.8及以上版本的rabbitmq ，你必须使用rabbitmq_prometheus插件。

作为文档state：

该插件以 Prometheus 文本格式在专用 TCP 端口上公开所有 RabbitMQ 指标。

要启用它，请运行：

rabbitmq-plugins enable rabbitmq_prometheus

要验证它是否在rabbitmq容器内运行：

curl -s localhost:15692/metrics | head -n 3

看看你是否得到指标作为回应。

对于 3.8 之前的版本，您必须使用prometheus_rabbitmq_exporter