kubenetes can'n not get nodes info after approve certificate from kubelet

Question

I install kubernetes with the way of manual. I can't get nodes info after master approve request of certificate from kubelet.

First, etcd kube-apiserver kube-controller-manager and kube-scheduler have been installed correctly. Execute commands kubectl get cs got the follow output:

[root@master cfg]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-2               Healthy   {"health":"true"}
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}

Then, I try to install kubelet on master node and a worker node. On master node execute command kubectl approve node-csr-V_FXPiKHAtqS_9GH27RCk6hPNWE0nF8bLSH6Ot7C360 and kubectl get csr :

[root@master cfg]# kubectl get csr
NAME                                                   AGE    SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-Tbrw3ia-c0D7kgIURIe_JnP1PCEdwf9XeATP0KcsyBg   111m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued
node-csr-V_FXPiKHAtqS_9GH27RCk6hPNWE0nF8bLSH6Ot7C360   131m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued

However, I can't get nodes from this cluster:

[root@master cfg]# kubectl get node
No resources found in default namespace.

Any body can take me a hand? Looking forward your help or something ideas to trace the problem. Thanks !

Answer 1

I had found the reason of this problem. Service of kubelet on master node and worker node don't really up. I got error from /var/log/message, follow:

Dec 11 21:21:20 worker1 systemd: Started Kubernetes Kubelet.
Dec 11 21:21:20 worker1 systemd: Starting Kubernetes Kubelet...
Dec 11 21:21:20 worker1 systemd: Started Kubernetes systemd probe.
Dec 11 21:21:20 worker1 systemd: Starting Kubernetes systemd probe.
Dec 11 21:21:20 worker1 kubelet: F1211 21:21:20.905947    3066 server.go:274] failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

which means docker's cgroup driver config is conflict with kubelet. After updated docker service file /usr/lib/systemd/system/docker.service , set --exec-opt native.cgroupdriver=cgroupfs :

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer

[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
          --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
          --default-runtime=docker-runc \
          --exec-opt native.cgroupdriver=cgroupfs \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          --init-path=/usr/libexec/docker/docker-init-current \
          --seccomp-profile=/etc/docker/seccomp.json \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY \
          $REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
KillMode=process

[Install]
WantedBy=multi-user.target

Then, restart docker and kubelet service, problem had been solved. Service kubelet work correctly.

Execute command kubelet get nodes , output follow:

[root@master ~]# kubectl get nodes
NAME      STATUS     ROLES    AGE    VERSION
master    NotReady   <none>   8s     v1.18.3
worker1   NotReady   <none>   112s   v1.18.3