Prometheus in k8s (metrics)

Question

I deploy prometheus in kubernetes on this manual

As a storage scheme was invented: Prometeus in kubernetes stores the metrics within 24 hours. Prometheus not in kubernetes stores the metrics in 1 week. A federation is set up between them.

Who faced with the fact that after removing the pods after a certain period of time (much less than 24 hours) metrics are missing on it.

Answer 1

This is perfectly normal if you do not have a persistent storage configured for your prometheus pod. You should use PV/PVC to define a stable place where you keep your prometheus data, otherwise if your pod is recreated, it starts with a clean slate.

Answer 2

PV/PVC needs dedicated storage servers in the cluster. If there is no money for storage servers, here is a cheaper approach:

1. Label a node:

    $ kubectl label nodes <node name> prometheus=yes 

2. Force all the prometheus pods to be created on the same labeled node by using nodeSelector :

    nodeSelector: prometheus: yes 

3. Create an emptyDir volume for each prometheus pod. An emptyDir volume is first created when the Prometheus pod is assigned to the labeled node and exists as long as that pod is running on that node and is safe across container crashes and pod restarts.

    spec: containers: - image: <prometheus image> name: <prometheus pod name> volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {} 

This approach makes all the Prometheus pods run on the same node with persistent storage for the metrics - a cheaper approach that prays the Prometheus node does not crash.