Monitoring Kubernetes persistent volumes usage with Prometheus

Question

I have a K8s 1.7 cluster using vSphere as persistent storage provider. I have also deployed Prometheus , node_exporter and kube-state-metrics .

I'm trying to find a way to monitor a persistent volume's usage using Prometheus . I have added custom labels to some PVs, eg. app=rabbitmq-0 , etc.

How can I combine kube_persistentvolume_labels with node_filesystem_size metrics so that I can query PV usage using my custom label?

PS.
I know that K8s 1.8 directly exposes these metrics from kubelet as mentioned in How to monitor disk usage of kubernetes persistent volumes? but currently a cluster upgrade is not an option.

Answer 1

Starting from (v1.3.0-rc.0 / 2018-03-23) in the kube-state-metrics , two metrics that can convert PersistentVolume and PersistenVolumeClaims labels to Prometheus labels accordingly were added:

kube_persistentvolume_labels
kube_persistentvolumeclaim_lables

To get more details about implementing aggregation of metrics based on labels, consider reading these articles:

• Aggregating Pod resource (CPU, memory) usage by arbitrary labels with Prometheus
• Exposing the software version to Prometheus

Answer 2

There are cases when hostpath cannot be mount to pod or node will not be accessible to a project (namespace) in that case node exporter cannot be deployed, in such cases volume_exporter would be useful.

You can just add it as a side car

      - name: volume-exporter
      image:  mnadeem/volume_exporter
      imagePullPolicy: "Always"
      args:
        - --volume-dir=prometheus:/prometheus
      ports:
      - name: metrics-volume
        containerPort: 9888
      volumeMounts:
      - mountPath: /prometheus
        name: prometheus-data
        readOnly: true

It will generate the metrics

        # HELP volume_bytes_free Free size of the volume/disk
        # TYPE volume_bytes_free gauge
        volume_bytes_free{volume_name="bin",volume_path="/bin"} 4.341569536e+10
        volume_bytes_free{volume_name="etc",volume_path="/etc"} 4.341569536e+10
        # HELP volume_bytes_total Total size of the volume/disk
        # TYPE volume_bytes_total gauge
        volume_bytes_total{volume_name="bin",volume_path="/bin"} 6.391887872e+10
        volume_bytes_total{volume_name="etc",volume_path="/etc"} 6.391887872e+10
        # HELP volume_bytes_used Used size of volume/disk
        # TYPE volume_bytes_used gauge
        volume_bytes_used{volume_name="bin",volume_path="/bin"} 2.050318336e+10
        volume_bytes_used{volume_name="etc",volume_path="/etc"} 2.050318336e+10
        # HELP volume_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which volume_exporter was built.
        # TYPE volume_exporter_build_info gauge
        volume_exporter_build_info{branch="",goversion="go1.15",revision="",version=""} 1
        # HELP volume_percentage_used Percentage of volume/disk Utilization
        # TYPE volume_percentage_used gauge
        volume_percentage_used{volume_name="bin",volume_path="/bin"} 32.07688208958619
        volume_percentage_used{volume_name="etc",volume_path="/etc"} 32.07688208958619

Refer this for more details

Disclaimer : I am the owner