轮换 Kubernetes 证书的最佳做法是什么

Question

Currently I am using a script to renew Kubernetes certificates before they expire. But this is a manual process. I have to monitor expiration dates carefully and run this script beforehand. What's the recommended way to update all control plane certificates automatically without updating control plane? Do kubelet's --rotate* flags rotate all components (eg controller) or it is just for kubelet? PS: Kubernetes cluster was created with kubeadm.

Answer 1

As of kubernetes 1.8 certificate rotation is added. You can read about it here, https://kubernetes.io/docs/tasks/tls/certificate-rotation/

Answer 2

Answering following question:

What's the recommended way to update all control plane certificates automatically without updating control plane

According to the k8s docs and best practices the best practice is to use "Automatic certificate renewal" with control plane upgrade:

Automatic certificate renewal

This feature is designed for addressing the simplest use cases; if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.

Note: It is a best practice to upgrade your cluster frequently in order to stay secure.

-- Kubernetes.io: Administer cluster: Kubeadm certs: Automatic certificate renewal

Why this is the recommended way:

From the best practices standpoint you should be upgrading your control-plane to patch vulnerabilities, add features and use the version that is currently supported.

Each control-plane upgrade will renew the certificates as described (defaults to true ):

• $ kubeadm upgrade apply --help

--certificate-renewal    Perform the renewal of certificates used by component changed during upgrades. (default true)

You can also check the expiration of the control-plane certificates by running:

• $ kubeadm certs check-expiration

[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 30, 2022 13:36 UTC   364d                                    no      
apiserver                  May 30, 2022 13:36 UTC   364d            ca                      no      
apiserver-etcd-client      May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   May 30, 2022 13:36 UTC   364d            ca                      no      
controller-manager.conf    May 30, 2022 13:36 UTC   364d                                    no      
etcd-healthcheck-client    May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
etcd-peer                  May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
etcd-server                May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
front-proxy-client         May 30, 2022 13:36 UTC   364d            front-proxy-ca          no      
scheduler.conf             May 30, 2022 13:36 UTC   364d                                    no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      May 28, 2031 13:36 UTC   9y              no      
etcd-ca                 May 28, 2031 13:36 UTC   9y              no      
front-proxy-ca          May 28, 2031 13:36 UTC   9y              no  

A side note!

kubelet.conf is not included in the list above because kubeadm configures kubelet for automatic certificate renewal.

From what it can be seen by default:

• Client certificates generated by kubeadm expire after 1 year.
• CA created by kubeadm are set to expire after 10 years.

---

There are other features that allows you to rotate the certificates in a "semi automatic" way.

You can opt for a manual certificate renewal with the:

• $ kubeadm certs renew

where you can automatically (with the command) renew the specified (or all) certificates:

• $ kubeadm certs renew all

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
 
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Please take a specific look on the output:

You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

As pointed, you will need to restart the components of your control-plane to use new certificate but remember:

• $ kubectl delete pod -n kube-system kube-scheduler-ubuntu will not work .

You will need to restart the docker container responsible for the component:

• $ docker ps | grep -i "scheduler"
• $ docker restart 8c361562701b (example)

8c361562701b   38f903b54010             "kube-scheduler --au…"   11 minutes ago      Up 11 minutes                k8s_kube-scheduler_kube-scheduler-ubuntu_kube-system_dbb97c1c9c802fa7cf2ad7d07938bae9_5
b709e8fb5e6c   k8s.gcr.io/pause:3.4.1   "/pause"                 About an hour ago   Up About an hour             k8s_POD_kube-scheduler-ubuntu_kube-system_dbb97c1c9c802fa7cf2ad7d07938bae9_0

---

As pointed in below link, kubelet can automatically renew it's certificate ( kubeadm configures the cluster in a way that this option is enabled):

• Kubernetes.io: Configure Certificate Rotation for the Kubelet
• Github.com: Kubernetes: Kubeadm: Issues: --certificate-renewal true doesn't renew kubelet.conf

Depending on the version used in your environment, this can be disabled. Currently in the newest version of k8s managed by kubeadm this option is enabled by default according to my knowledge.

---

Please keep in mind that before you start with any kubernetes node/control plane/update/upgrade to read "Urgent Upgrade Notes" specific to your k8s version (example):

• Github.com: Kubernetes: CHANGELOG: 1.21: Urgent upgrade nodes

---

Defining the automatic way of certificate rotation could go in either way but you can use already mentioned commands to automate this process. You would need to create a script (which you already have) that would be put in cron that would fire after some time and renew them.