使用证书续订后 kubelet.service 失败

Question

我试图更新过期的证书，我按照以下步骤操作，kubectl 服务开始失败。 我是 kubernetes 的新手，请帮助我。

# kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Nov 11, 2020 12:52 UTC   <invalid>       no
apiserver                  Nov 11, 2020 12:52 UTC   <invalid>       no
apiserver-etcd-client      Nov 11, 2020 12:52 UTC   <invalid>       no
apiserver-kubelet-client   Nov 11, 2020 12:52 UTC   <invalid>       no
controller-manager.conf    Nov 11, 2020 12:52 UTC   <invalid>       no
etcd-healthcheck-client    Nov 11, 2020 12:52 UTC   <invalid>       no
etcd-peer                  Nov 11, 2020 12:52 UTC   <invalid>       no
etcd-server                Nov 11, 2020 12:52 UTC   <invalid>       no
front-proxy-client         Nov 11, 2020 12:52 UTC   <invalid>       no
scheduler.conf             Nov 11, 2020 12:52 UTC   <invalid>       no

# kubeadm alpha certs renew all
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

 # kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Nov 17, 2021 05:49 UTC   364d            no
apiserver                  Nov 17, 2021 05:49 UTC   364d            no
apiserver-etcd-client      Nov 17, 2021 05:49 UTC   364d            no
apiserver-kubelet-client   Nov 17, 2021 05:49 UTC   364d            no
controller-manager.conf    Nov 17, 2021 05:49 UTC   364d            no
etcd-healthcheck-client    Nov 17, 2021 05:49 UTC   364d            no
etcd-peer                  Nov 17, 2021 05:49 UTC   364d            no
etcd-server                Nov 17, 2021 05:49 UTC   364d            no
front-proxy-client         Nov 17, 2021 05:49 UTC   364d            no
scheduler.conf             Nov 17, 2021 05:49 UTC   364d            no

 :~> mkdir -p $HOME/.kube
 :~> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 :~> sudo chown $(id -u):$(id -g) $HOME/.kube/config

 :~> sudo systemctl daemon-reload
 :~> sudo systemctl stop kubelet
 :~> sudo systemctl start kubelet
 :~> sudo systemctl enable kubelet
 :~> sudo systemctl stop docker 
 :~> sudo systemctl start docker

  :~> kubectl get pods
The connection to the server 10.xx.xx.74:6443 was refused - did you specify the right host or port?
 
# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean"GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server 10.xx.xx.74:6443 was refused - did you specify the right host or port?

Kubectl 状态：

 #  systemctl status kubelet
  kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Tue 2020-11-17 08:18:20 UTC; 1s ago
     Docs: https://kubernetes.io/docs/
  Process: 1452 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, sta
 Main PID: 1452 (code=exited, status=255)

Nov 17 08:18:20 c536gocrb systemd[1]: Unit kubelet.service entered failed state.
Nov 17 08:18:20 c536gocrb systemd[1]: kubelet.service failed.

尝试将环境变量添加到 admin 10.kubeadm.conf

Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"

kubeadm 配置文件：

 cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"


# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Answer 1

问题已解决。 按照https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration 中的建议替换 kubelet.conf 中的证书数据后

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ==
    server: https://xx.x.x.x.:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:cmaster
  name: system:node:cmaster@kubernetes
current-context: system:node:cmaster@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:cmaster
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem