Kubelet Master stays in KubeletNotReady because of cni missing

Question

Kubelet has been initialized with pod network for Calico :

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --image-repository=someserver

Then i get calico.yaml v3.11 and applied it :

sudo kubectl --kubeconfig="/etc/kubernetes/admin.conf" apply -f calico.yaml

Right after i check on the pod status :

sudo kubectl --kubeconfig="/etc/kubernetes/admin.conf" get nodes
NAME              STATUS     ROLES    AGE     VERSION
master-1   NotReady   master   7m21s   v1.17.2

on describe i've got cni config unitialized, but i thought that calico should have done that ?

MemoryPressure   False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 21 Feb 2020 10:14:24 +0100   Fri, 21 Feb 2020 10:09:00 +0100   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

In fact i have nothing under /etc/cni/net.d/ so it seems it forgot something ?

ll /etc/cni/net.d/
total 0

sudo kubectl --kubeconfig="/etc/kubernetes/admin.conf" -n kube-system get pods
    NAME                                       READY   STATUS                  RESTARTS   AGE
calico-kube-controllers-5644fb7cf6-f7lqq   0/1     Pending                 0          3h
calico-node-f4xzh                          0/1     Init:ImagePullBackOff   0          3h
coredns-7fb8cdf968-bbqbz                   0/1     Pending                 0          3h24m
coredns-7fb8cdf968-vdnzx                   0/1     Pending                 0          3h24m
etcd-master-1                       1/1     Running                 0          3h24m
kube-apiserver-master-1            1/1     Running                 0          3h24m
kube-controller-manager-master-1    1/1     Running                 0          3h24m
kube-proxy-9m879                           1/1     Running                 0          3h24m
kube-scheduler-master-1             1/1     Running                 0          3h24m

As explained i'm running through a local repo and journalctl says :

 kubelet[21935]: E0225 14:30:54.830683   21935 pod_workers.go:191] Error syncing pod cec2f72b-844a-4d6b-8606-3aff06d4a36d ("calico-node-f4xzh_kube-system(cec2f72b-844a-4d6b-8606-3aff06d4a36d)"), skipping: failed to "StartContainer" for "upgrade-ipam" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://repo:10000/v2/calico/cni/manifests/v3.11.2: no basic auth credentials"
 kubelet[21935]: E0225 14:30:56.008989   21935 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Feels like it's not only CNI the issue

Answer 1

Core DNS pod will be pending and master will be NotReady till calico pods are successfully running and CNI is setup properly.

It seems to be network issue to download calico docker images from docker.io. So you can pull calico images from docker.io and and push it to your internal container registry and then modify the calico yaml to refer that registry in images section of calico.yaml and finally apply the modified calico yaml to the kubernetes cluster.

Answer 2

So the issue with Init:ImagePullBackOff was that it cannot apply image from my private repo automatically. I had to pull all images for calico from docker. Then i deleted the calico pod it's recreate itself with the newly pushed image

sudo docker pull private-repo/calico/pod2daemon-flexvol:v3.11.2
sudo docker pull private-repo/calico/node:v3.11.2
sudo docker pull private-repo/calico/cni:v3.11.2
sudo docker pull private-repo/calico/kube-controllers:v3.11.2

sudo kubectl -n kube-system delete po/calico-node-y7g5

After that the node re-do all the init phase and :

sudo kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5644fb7cf6-qkf47   1/1     Running   0          11s
calico-node-mkcsr                          1/1     Running   0          21m
coredns-7fb8cdf968-bgqvj                   1/1     Running   0          37m
coredns-7fb8cdf968-v85jx                   1/1     Running   0          37m
etcd-lin-1k8w1dv-vmh                       1/1     Running   0          38m
kube-apiserver-lin-1k8w1dv-vmh             1/1     Running   0          38m
kube-controller-manager-lin-1k8w1dv-vmh    1/1     Running   0          38m
kube-proxy-9hkns                           1/1     Running   0          37m
kube-scheduler-lin-1k8w1dv-vmh             1/1     Running   0          38m