无法加载 kubelet 配置文件

Question

hy folks

after updating my server, I can't restart kubernetes.

Feb  6 10:34:26 chgvas99 kubelet: F0206 10:34:26.662744   27634 server.go:189] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Feb  6 10:34:26 chgvas99 systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Feb  6 10:34:26 chgvas99 systemd: Unit kubelet.service entered failed state.
Feb  6 10:34:26 chgvas99 systemd: kubelet.service failed.

i checked on the directory and indeed there is no config.yaml i've the same error on my nodes i cant restart them

server : 3.10.0-957.5.1.el7.x86_64

kubernetes : Major:"1", Minor:"13", GitVersion:"v1.13.3" GoVersion:"go1.11.5"

Answer 1

I would recommend runninng 'kubeadm-init' to reinitialise the cluster. Also please make sure you '/var' directory is not full. Please see this link for more information about 'kubeadm init' command.

Answer 2

You're using kubeadm - So The fact that /var/lib/kubelet/config.yaml is empty is probably related to the worker node not being joined to the cluster.

This might be related to networking issues - but lets try step by step:

1 ) Create a valid token for the worker node to join the cluster:
Run: sudo kubeadm token create --print-join-command --v=5 and make sure you receive an output command like:

kubeadm join <master-node-ip>:6443 --token aa334.. --discovery-token-ca-cert-hash sha256:..

2 ) Run the provided command in the worker node.

3 ) If Everything is OK - the /var/lib/kubelet/config.yaml should be populated and the status of sudo systemctl status kubelet should look good.

4 ) If you can an error - try running the ame join command with --v=5 - you'll probably see some networking issues.

4.A ) If you got an error like dial tcp <master-ip>:6443: connect: no route to host - make sure that you have a communication between your nodes - run curl <master-node-ip>:6443 from worker node - you'll probably get the same no route error.
Go to the master node and open the 6443 port (I'll assume you're working on secured private network) and try the connectivity again.

4.B ) If opening port in master succeed and you're able to curl from worker to master you should receive a response from the API server like: Client sent an HTTP request to an HTTPS server .

5 ) If curl succeed but you're still facing connectivity problems try:

5.A ) Comparing the .kube/config files of master and worker nodes - make sure the IP of the API server is correct.

5.B ) Make sure you enabled bridge networking mode on all nodes: sudo sysctl net.bridge.bridge-nf-call-iptables=1 .

5.C ) Make sure you have an SDN solution likecalico , flannel or weave and that you see that the relevant kube-system pods are running:

$kubectl -n kube-system get pods
NAME                                      READY   STATUS    RESTARTS   AGE
coredns-f9fd979d6-lpdlc                   1/1     Running   2          7d12h
coredns-f9fd979d6-vcs7g                   1/1     Running   2          7d12h
etcd-master-node-k8s                      1/1     Running   2          7d12h
kube-apiserver-master-node-k8s            1/1     Running   2          7d12h
kube-controller-manager-master-node-k8s   1/1     Running   2          7d12h
kube-proxy-kh2lc                          1/1     Running   2          7d12h
kube-proxy-lfmc4                          1/1     Running   0          4m36s
kube-scheduler-master-node-k8s            1/1     Running   2          7d12h
weave-net-59r5b                           2/2     Running   6          7d11h <-- Here 
weave-net-c44d6                           2/2     Running   1          4m36s <-- Here

6 ) If nothing works - try running kubeadm reset on the worker node.

Answer 3

My Env: 3 master and forefront loadbalancer while initializing cluster

Had to adjust my load balancer and take out other master nodes and "kubeadm init" and issue was gone , Network error possibly .

Then readd all the other master nodes.

Answer 4

Had a similar issue where got that error on control join command. It ended up being the load balancer ip assigned to the joining control as secondary from a previous installation (which has nothing running behind it). Run ip a and make sure that the joining node does not have secondary load balancer ip.

echo y | kubeadm reset || true
rm -rf /etc/cni/net.d || true
rm -rf /var/lib/etcd || true
rm -rf ~/.kube || true
ip address delete {{ k8s_apiserver_vip_cidr }} dev {{ k8s_interface }}
kubeadm join ...