更新 k8s Prometheus 操作员的配置以添加抓取目标

Question

I have an existing deployment of prometheus operator on my k8s cluster. Now, I want to add a scrape target for a custom exporter I created. I created my prometheus.yaml file, but can't find how to apply this to the existing prometheus operator.

Answer 1

I have no idea how and where you create/modify your prometheus.yaml file, but will show you the convenient way of managing it.

First of all you I would recommend you store prometheus configuration file prometheus.yaml as a configmap. This is super useful plus changes you will do in CM will be automatically without your involving spreading/propagating to the pod that work with/consume this CM.

So after you make changes with new scrapes in CM - it will take some time to propagate changes. Total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period (1 minute by default) + ttl of ConfigMaps cache (1 minute by default) in kubelet.

Now time to make changes live.. Prometheus has an option to reload its configuration on flight. You have 2 options how to do that. More info you can find opening Reloading Prometheus' Configuration url.

I will stop here only on one solution: You can send a HTTP POST to the Prometheus web server:

curl -X POST http://localhost:9090/-/reload

Note that as of Prometheus 2.0, the --web.enable-lifecycle command line flag must be passed for HTTP reloading to work. If the reload is successful Prometheus will log that it has updated its targets:

INFO[0248] Loading configuration file prometheus.yml source=main.go:196
INFO[0248] Stopping target manager... source=targetmanager.go:203
INFO[0248] Target manager stopped. source=targetmanager.go:216
INFO[0248] Starting target manager... source=targetmanager.go:114

And cherry on cake for you:)

There is a great Prometheus, ConfigMaps and Continuous Deployment that explain how to monitor Prometheus with prometheus(Maybe it will be applicable somehow in your another question). The main stuff I want to show you is that you can automate POST request.

So basically you need a tiny sidecar container that will check CM changes and send POST in case of new changes. This sidecar should be in the same pod with prometheus

Copy pasting example from article here for future reference

 spec:
  containers:
  - name: prometheus
    ...
    volumeMounts:
    - name: config-volume
      mountPath: /etc/prometheus
  - name: watch
    image: weaveworks/watch:master-5b2a6e5
    imagePullPolicy: IfNotPresent
    args: ["-v", "-t", "-p=/etc/prometheus", "curl", "-X", "POST", "--fail", "-o", "-", "-sS", "http://localhost:80/-/reload"]
    volumeMounts:
    - name: config-volume
      mountPath: /etc/prometheus
  volumes:
    - name: config-volume
      configMap:
        name: prometheus-config

Answer 2

Documentation for the operator: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md

But you have to use additionalScrapeConfigs Secret :

    additionalScrapeConfigsSecret:
      enabled: true
      name: additional-scrape-configs
      key: prometheus-additional.yaml

Else you get the Error cannot unmarshal.!map into []yaml.MapSlice

Here is a better documentation: https://github.com/prometheus-community/helm-charts/blob/8b45bdbdabd9b54766c4beb3c562b766b268a034/charts/kube-prometheus-stack/values.yaml#L2691

According to this, you could add scrape configs without packaging into a secret like this:

additionalScrapeConfigs: |
  - job_name: "prometheus"
  static_configs:
  - targets: ["localhost:9090"]