Kubernetes 上没有持久的 Prometheus 指标

Question

我正在从托管在 Kubernetes 上的 uwsgi 应用程序收集 Prometheus 指标，删除 pod 后不会保留这些指标。 Prometheus 服务器托管在同一个 kubernetes 集群上，我为它分配了一个持久存储。

即使在删除后，我如何保留 pod 中的指标？

Prometheus 部署 yaml：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--storage.tsdb.retention=2200h"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          persistentVolumeClaim:
            claimName: azurefile
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus
  name: prometheus
spec:
  type: LoadBalancer
  loadBalancerIP: ...
  ports:
    - port: 80
      protocol: TCP
      targetPort: 9090
  selector:
    app: prometheus

应用部署yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api-app
  template:
    metadata:
      labels:
        app: api-app
    spec:
      containers:
      - name: nginx
        image: nginx
        lifecycle:
          preStop:
            exec:
              command: ["/usr/sbin/nginx","-s","quit"]
        ports:
          - containerPort: 80
            protocol: TCP
        resources:
          limits:
            cpu: 50m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 50Mi
        volumeMounts:
          - name: app-api
            mountPath: /var/run/app
          - name: nginx-conf
            mountPath: /etc/nginx/conf.d
      - name: api-app
        image: azurecr.io/app_api_se:opencv
        workingDir: /app
        command: ["/usr/local/bin/uwsgi"]
        args:
          - "--die-on-term"
          - "--manage-script-name"
          - "--mount=/=api:app_dispatch"
          - "--socket=/var/run/app/uwsgi.sock"
          - "--chmod-socket=777"
          - "--pyargv=se"
          - "--metrics-dir=/storage"
          - "--metrics-dir-restore"
        resources:
          requests:
            cpu: 150m
            memory: 1Gi
        volumeMounts:
          - name: app-api
            mountPath: /var/run/app
          - name: storage
            mountPath: /storage
      volumes:
        - name: app-api
          emptyDir: {}
        - name: storage  
          persistentVolumeClaim:
            claimName: app-storage
        - name: nginx-conf
          configMap:
            name: app
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "test"
        effect: "NoSchedule"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: api-app
  name: api-app
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: api-app

Answer 1

使用此卷配置，当您发布 Pod 时，它将被删除。 您基本上是在寻找 PersistentVolumne、文档和示例。

还要检查PersistentVolumeClaim 。

Answer 2

您的问题是用于部署 Prometheus 的控制器类型错误。
在这种情况下，部署控制器是错误的选择（它适用于无状态应用程序，不需要在重新调度的 Pod 之间维护任何持久性标识符——比如持久性数据）。

您应该切换到StatefulSet那种*，如果你需要跨吊舱（重新）调度数据的持久性（指标普罗米修斯刮）。

*这是默认情况下使用prometheus-operator部署 Prometheus 的方式。