Prometheus auto discovery K8s

Question

Can someone guide the configuration for auto discover for K8s. The Prometheus server is outside of the cluster. I tried Service Discovery With Kubernetes and someone mentioned in this discussion

I'm not yet a K8s expert enough to explain all the details here, but fundamentally it's perfectly possible to run Prometheus outside of the cluster (and required for things like redundant cross-cluster meta-monitoring). Cf. the in_cluster config option in http://prometheus.io/docs/operating/configuration/#kubernetes-sd-configurations-kubernetes_sd_config . You need to jump through certificate hoops if you run it outside.

So, I made a simple configuration

  - job_name: 'kubernetes'

    kubernetes_sd_configs:
      -
        # The API server addresses. In a cluster this will normally be
        # `https://kubernetes.default.svc`. Supports multiple HA API servers.
        api_servers:
          - https://xxx.xx.xx.xx

        # Run in cluster. This will use the automounted CA certificate and bearer
        # token file at /var/run/secrets/kubernetes.io/serviceaccount/ in the pod.
        in_cluster: false

        # Optional HTTP basic authentication information.
        basic_auth:
        username: prometheus
        password: secret

        # Retry interval between watches if they disconnect.
        retry_interval: 5s

Getting unknown fields in kubernetes_sd_config: api_servers, in_cluster, retry_interval" or some other indentation errors

In sample configuration , they mentioned ca_file: . How to get that certificate file from K8s or is there any way to specify K8s config file(~/.kube/config)

Answer 1

By digging though the source code I figured out, that Prometheus always uses the in cluster config, if no api_server is provided in the config ( discovery/kubernetes/kubernetes.go#L90-L96 ).

Somehow the docs don't say anything about the Kubernetes configuration parameters, but the source code does ( config/config.go#L1026-L1037 ). Therefore there is not list named api_servers , but a single parameter named api_server .

So your config should look like this (untested):

  - job_name: 'kubernetes'

    kubernetes_sd_configs:
      -
        # The API server addresses. In a cluster this will normally be
        # `https://kubernetes.default.svc`. Supports multiple HA API servers.
        api_server: https://xxx.xx.xx.xx

        # Optional HTTP basic authentication information.
        basic_auth:
          username: prometheus
          password: secret

        # specify the CA
        tls_config:
          ca_file: /path/to/ca.crt
          ## If the actual CA file isn't available you need to disable verification:
          # insecure_skip_verify: true

I don't know where the retry_interval parameter comes from, but AFAIK this isn't a Kubernetes config parameter and it's also not part of the Prometheus config.

Answer 2

With help of @svenwltr answer I have create docker image which we can launch in K8s cluster. Check my repo

Answer 3

How to retrieve that file depends on your cluster setup

How to get that certificate file from K8s

By default, kubernetes stores the client CA certificate file at /etc/kubernetes/pki/ca.crt and also at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt in a ConfigMap used by kubeconfig .