Prometheus cannot scrape external etcd

Question

I have Kubernetes cluster running on AWS instances, and prometheus running inside kubernetes for monitoring. There are three etcd servers running external to kubernetes, and I am trying to use prometheus to monitor etcd health.

Prometheus is deployed as a statefulset, and has metrics for kubelet, node-exporters, and itself. However, I cannot get any metrics from etcd.

Here is the relevant part of prometheus's config:

apiVersion: v1
kind: ConfigMap
metadata:
   name: prometheus
   namespace: monitoring
   data:
   prometheus.yml: |-
global:
  scrape_interval: 30s
  evaluation_interval: 30s

rule_files:
- /etc/alertmanager/*.rules

scrape_configs:

- job_name: etcd
  scheme: https
  static_configs:
  - targets: ['x.x.x.x:2379']
  tls_config:
     ca_file: /etc/etcd/ssl/ca.pem
     cert_file: /etc/etcd/ssl/client.pem
     key_file: /etc/etcd/ssl/client-key.pem
     insecure_skip_verify: true

- job_name: kubelets
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

This is the error I am getting in Prometheus's dashboard:

Get https://x.x.x.x.:2379/metrics: x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

The cert is self-signed, but shouldn't "insecure_skip_verify" take care of that?

Answer 1

To eliminate etcd issues, if you're using etcd3, you can use the following arguments with the etcd client etcdctl and interact with the etcd server using the steps in https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md . If it works without errors, I'd say this is a prometheus issue for not honoring the insecure_skip_verify: true configuration.

--insecure-skip-tls-verify=true   skip server certificate verification
--insecure-transport=true         disable transport security for client connections