Kubernetes kubelet-certificate-authority on premise with kubespray causes certificate validation error for master node

Question

I'm setting up a k8s cluster on premise using kubespray.

I'm trying to harden the kube.netes cluster using CIS Benchmark documentation.

For the --kubelet-certificate-authority argument I set up the TLS connection between the apiserver and kubelets. Then, I edited the API server pod specification file /etc/kube.netes/manifests/kube-apiserver.yaml on the master node and set the --kubelet-certificate-authority parameter for the certificate authority like this: --kubelet-certificate-authority=/etc/kube.netes/ssl/apiserver.crt

But with that I'm not longer able top deploy pods (using helm), having the known error:

[centos@infra-vm ~]$ helm list Error: forwarding ports: error upgrading connection: error dialing backend: x509: cannot validate certificate for 192.168.33.143 because it doesn't contain any IP SANs

Where 192.168.33.143 is the master node IP address.

I've checked the above certificate autority and is has IP SANs:

So I really can't figure out where the issue comes from.

[centos@infra-vm ~]$ kubectl get pod --namespace kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7555c9885c-tjz78    1/1     Running   0          3d21h
calico-node-2p4p4                           1/1     Running   0          3d21h
calico-node-4rhzj                           1/1     Running   0          3d21h
coredns-56bc6b976d-wrxsl                    1/1     Running   0          3d21h
coredns-56bc6b976d-zlvxb                    1/1     Running   0          3d21h
dns-autoscaler-5fc5fdbf6-sl6gg              1/1     Running   0          3d21h
kube-apiserver-cpu-node0                    1/1     Running   0          3d21h
kube-controller-manager-cpu-node0           1/1     Running   0          3d21h
nvidia-device-plugin-daemonset-1.12-zj82x   1/1     Running   0          3d20h
tiller-deploy-677fbf76bb-hcgtw              1/1     Running   0          3d21h

[centos@infra-vm ~]$ kubectl logs tiller-deploy-677fbf76bb-hcgtw  --namespace kube-system
Error from server: Get https://192.168.33.143:10250/containerLogs/kube-system/tiller-deploy-677fbf76bb-hcgtw/tiller: x509: cannot validate certificate for 192.168.33.143 because it doesn't contain any IP SANs
[centos@infra-vm ~]$

Could one try to help figure out what is going on?

Answer 1

First of all /etc/kube.netes/ssl/apiserver.crt is not a valid CA certificate. CA would have:

    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment, Certificate Sign

Notice Certificate Sign extension that allows for signing certificates.

---

You are seeing this error: cannot validate certificate for 192.168.33.143 because it doesn't contain any IP SANs because kubelet is using self signed certificates to serve https traffic on port 10250 and you are using invalid certificate to validate it.

---

So what should you do to make it work??

• Use /etc/kube.netes/ssl/ca.crt to sign new certificate for kubelet with valid IP SANs.
• Set --kubelet-certificate-authority=/etc/kube.netes/ssl/ca.crt (valid CA).
• In /var/lib/kubelet/config.yaml (kubelet config file) set tlsCertFile and tlsPrivateKeyFile to point to newly created kubelet crt and key files.