Kubernetes on worker node - kubelet.service not starting

Question

I am trying to setup a new worker-node on CentOS-7.9 with following commands.

# setenforce 0
# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

# firewall disabled already.
# swapoff -a

# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

# yum install kubeadm -y 
# systemctl enable kubelet
# systemctl start kubelet

But kubelet service status shows below error.

# systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2020-12-02 16:49:22 IST; 3s ago
     Docs: https://kubernetes.io/docs/
  Process: 4442 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 4442 (code=exited, status=255)

Dec 02 16:49:22 k8s-node-01 systemd[1]: Unit kubelet.service entered failed state.
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_output/dockerized/go/src/...81 +0x4f
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_output/dockerized/go/src/...58 +0x8a
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: goroutine 47 [select]:
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.(*worker).start(0xc000050be0)
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_output/dockerized/go/src/...4 +0x105
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: created by k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.init.0
Dec 02 16:49:22 k8s-node-01 kubelet[4442]: /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_output/dockerized/go/src/...32 +0x57
Dec 02 16:49:22 k8s-node-01 systemd[1]: kubelet.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

I have below Kube.netes & docker version installed.

# kubelet --version
Kubernetes v1.19.4

# docker --version
Docker version 19.03.14, build 5eb3275d40

Also tried to join but even this fails.

# kubeadm join 65.66.67.68:6443 --token tu7qvt.1rfzhnxevg8m792h     --discovery-token-ca-cert-hash sha256:48109668a4eadfs3c0c13a04d24a99bd82ff2eredefab6be6b78aadeead358074ee
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 65.66.67.55:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 65.66.67.55:10248: connect: connection refused.
error execution phase kubelet-start: error uploading crisocket: timed out waiting for the condition
To see the stack trace of this error execute with --v=5 or higher

With -v=9 option:-

# kubeadm join 65.66.67.68:6443 --token tu7qvt.1rfzhnxevg8m792h     --discovery-token-ca-cert-hash sha256:48109668a4eadfs3c0c13a04d24a99bd82ff2eredefab6be6b78aadeead358074ee -v=9

I1203 10:25:29.374052   11716 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.19.4 (linux/amd64) kubernetes/d360454" 'https://65.66.67.68:6443/api/v1/nodes/k8s-node-01?timeout=10s'
I1203 10:25:29.376358   11716 round_trippers.go:443] GET https://65.66.67.68:6443/api/v1/nodes/k8s-node-01?timeout=10s 404 Not Found in 2 milliseconds
I1203 10:25:29.376406   11716 round_trippers.go:449] Response Headers:
I1203 10:25:29.376411   11716 round_trippers.go:452]     Content-Type: application/json
I1203 10:25:29.376415   11716 round_trippers.go:452]     Content-Length: 192
I1203 10:25:29.376419   11716 round_trippers.go:452]     Date: Thu, 03 Dec 2020 04:55:29 GMT
I1203 10:25:29.376423   11716 round_trippers.go:452]     Cache-Control: no-cache, private
I1203 10:25:29.376443   11716 request.go:1097] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes \"k8s-node-01\" not found","reason":"NotFound","details":{"name":"k8s-node-01","kind":"nodes"},"code":404}
timed out waiting for the condition
error uploading crisocket

What could be the wrong in installation? Any direction would be helpful.

Answer 1

Node has joined the cluster after commenting the entries from /etc/resolv.conf file then once node has joined to the cluster successfully again Un-commented . Now on my master all the namespaces and nodes are running fine.