Kubernetes 集群外的 Prometheus

Question

I'm trying to configure Prometheus outside Kubernetes Cluster.

Below is my Prometheus config.

- job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://10.0.4.155:6443
  scheme: https
  tls_config:
    insecure_skip_verify: true
  basic_auth:
      username: kube
      password: Superkube01
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

These is how it looks:

root@master01:~# kubectl cluster-info 
Kubernetes master is running at https://10.0.4.155:6443

root@master01:~# kubectl get endpoints 
NAME                 ENDPOINTS                                         AGE
kubernetes           10.0.4.103:6443,10.0.4.138:6443,10.0.4.155:6443   11h
netchecker-service   10.2.0.10:8081                                    11h
root@master01:~# 

But, when starting Prometheus, i'm getting below error.

level=error ts=2018-05-29T13:55:08.171451623Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:270: Failed to list *v1.Pod: Get https://10.0.4.155:6443/api/v1/pods?resourceVersion=0: x509: certificate signed by unknown authority"

Could anyone please tell me, what wrong i'm doing here?

Thanks, Pavanasam R

Answer 1

The error indicates that Prometheus is using a different certificate to sign its metric collection request than the one expected by your apiserver.

You really need to format your code in a code block so we can see the yaml formatting. kubernetes_sd_configs seems to be the wrong home for insecure_skip_verify and basic_auth according to this link . Might want to move them and try scraping again.

Answer 2

As of now your insecure_skip_verify is a part of kubernetes_sd_configs:. Add it in api_server context as well.

kubernetes_sd_configs:
- api_server: https://<ip>:6443
  role: node
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true

Answer 3

in order to access kubernetes api endpoint you need to authenticate the client either through basic_auth, bearer_token, tls_config . please go through this , it will be helpful.