如何將 scrape_configs 文件添加到 Prometheus

Question

我已經使用 rancher-monitoring Helm 圖表在cattle-monitoring-system命名空間中部署了 Prometheus、Alertmanager、Grafana 等。 在此處和 此處查看文檔。 此命名空間中的所有內容都運行正常。

現在我正在嘗試使用清單文件在另一個名為indigo的命名空間中部署一個額外的 Prometheus 實例。 不幸的是，配置 scrape_configs 文件不會加載。 我通過運行命令kubectl exec -it prometheus-indigo-0 -c prometheus -n indigo -- cat /etc/prometheus/config_out/prometheus.env.yaml檢查了這一點。 在另一個 Prometheus 實例中運行相同的命令將返回所有的 scrape_configs 配置。

我的deployment.yml文件如下所示：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monitoring-rancher-monitor-prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: indigo
  namespace: $NAMESPACE
spec:
  serviceAccountName: prometheus
  additionalScrapeConfigs:
    name: prometheus-scrape-configs-secret
    key: prometheus-scrape-configs.yml
  resources:
    requests:
      memory: 400Mi
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-indigo
  namespace: $NAMESPACE
spec:
  type: NodePort
  ports:
  - name: web
    nodePort: 30900
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    prometheus: indigo

prometheus-scrape-configs.yml文件如下所示：

global:
  evaluation_interval: 30s
  scrape_interval: 30s
scrape_configs:
- job_name: blackbox # To get metrics about the exporter itself
  metrics_path: /metrics
  static_configs:
    - targets:
      - .....

- job_name: blackbox-http # To get metrics about the exporter’s targets
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - .....
      labels:
        env: elise
    - targets:
      - .....
      labels:
        env: osb
    - targets:
      - .....
      labels:
        env: itp
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: .....

最后，在Makefile中創建prometheus-scrape-configs-secret ：

include ../../../common.mk
include ../../environments/$(ENVIRONMENT).mk

NAMESPACE ?= indigo

.PHONY: deploy
deploy: \
    init \
    deploy-monitoring

.PHONY: init
init:

.PHONY: deploy-monitoring
deploy-monitoring:
    kubectl create secret generic prometheus-scrape-configs-secret \
    -n $(NAMESPACE) --from-file=prometheus-scrape-configs.yml

    envsubst < $(ENVIRONMENT)-deployment.yml | kubectl apply -f -

.PHONY: delete
delete: delete-monitoring

.PHONY: delete-monitoring
delete-monitoring:
    kubectl delete secret prometheus-scrape-configs-secret -n $(NAMESPACE)
    envsubst < $(ENVIRONMENT)-deployment.yml | kubectl delete -f -

那么有人知道為什么scrape_configs文件無法正確加載嗎？

編輯：

運行命令kubectl describe pod prometheus-indigo-0 -n indigo后，我注意到下面的這個錯誤。 我還注意到在部署這個 Prometheus 實例后它會出現一個錯誤，立即重新啟動，然后它正在運行..

State:       Running
  Started:   Tue, 14 Jun 2022 18:01:23 +0200
Last State:  Terminated
  Reason:    Error
  Message:   ts=2022-06-14T16:01:19.743Z caller=main.go:450 level=error msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"

Answer 1

多虧了這個文檔，我設法解決了這個問題。 我忘記創建一個通過標簽引用blackbox-exporter的 ServiceMonitor。 最終，您需要使用serviceMonitorSelector將其鏈接到 Kubernetes Prometheus 對象中。 您可以在此處查看有關如何配置serviceMonitorSelector的更多信息