Dynamic targets for Prometheus in Kubernetes?

Question

In my docker setup, I maintain targets.json file which is dynamically updated with targets to probe. The file starts empty but is appended with targets during some use case.

sample targets.json

[
  {
    "targets": [
      "x.x.x.x"
    ],
    "labels": {
      "app": "testApp1"
    }
  },
  {
    "targets": [
      "x.x.x.x"
    ],
    "labels": {
      "app": "testApp2"
    }
  }
]

This file is then provided to prometheus configuration as file_sd_configs . Everything works fine, targets get added to targets.json file due to some event in application and prometheus starts monitoring along with blackbox for health checks.

scrape_configs:
  - job_name: 'test-run'
    metrics_path: /probe
    params:
      module: [icmp]
    file_sd_configs:
      - files:
        - targets.json
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

Inside my node.js application I am able to append data to targets.json file, but now I trying to replicate this in Kube.netes on minikube. I tried adding in ConfigMap as following and it works, but I dont want to populate targets in configuration, but rather maintain a json file.

Can this be done using Persistent Volumes? The pod running Prometheus will always read the targets file and pod running application will write to targets file.

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-cm
data:
  targets.json: |-
    [
      {
        "targets": [
          "x.x.x.x"
        ],
        "labels": {
          "app": "testApp1"
        }
      }
    ]

Simply, what strategy in Kube.netes is recommended to so that one pod can read a json file and another pod can write to that file.

Answer 1

In order to achieve your goal you need to usePVC :

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (eg, can be mounted once read/write or many times read-only).

The json file needs to be persisted if one pod has to write to it and another one to read it. There is an official guide describing that concept in steps:

• Create a PersistentVolume

• Create a PersistentVolumeClaim

• Create a Pod that uses your PersistentVolumeClaim as a volume

I also recommend reading this: Create ReadWriteMany PersistentVolumeClaims on your Kube.netes Cluster as a supplement.