Kubernetes V1.11.0 HA cluster CoreDNS PODS not coming up

Question

I am trying to create Kubernetes V1.11.0 HA cluster in Centos 7 server. I have IPV6 disabled on my host.

https://kubernetes.io/docs/setup/independent/high-availability/

Based on these steps added all three master, coreDNS POD is not coming up, its keep logging below error in /var/log/messages file.

kubectl get pods -n kube-system
NAME                                READY     STATUS              RESTARTS   AGE
coredns-78fcdf6894-8t7bt            0/1       ContainerCreating   0          53m
coredns-78fcdf6894-plmll            0/1       ContainerCreating   0          53m

error message:

Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531244   10041 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531286   10041 kuberuntime_sandbox.go:56] CreatePodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531298   10041 kuberuntime_manager.go:646] createPodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531358   10041 pod_workers.go:186] Error syncing pod 7caa3f29-89c3-11e8-aa0f-00505693ca30 ("coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)"), skipping: failed to "CreatePodSandbox" for "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)\" failed: rpc error: code = Unknown desc = [failed to set up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to set up pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to teardown pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: failed to get IP addresses for \"eth0\": <nil>]"
Jul 17 10:09:28 master03 kubelet: W0717 10:09:28.636173   10041 cni.go:243] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "88183d8a24dcafca5b04e3a6454e83aca1417a6934f50cd52b3e6fffbae4f272"
Jul 17 10:09:28 master03 kernel: XFS (dm-18): Mounting V4 Filesystem

How to fix this error message and bring the coreDNS PODs?

Thanks

Answer 1

There is an issue on Coreos Github that describes exactly your case. And the suggestion to solve the issue was:

squeed : You are using an old version of CNI, which does not work on systems with ipv6 disabled. Please re-enable ipv6 or upgrade CNI.

And regarding the related issue on CNI Github he also mentioned:

squeed : As I already explained, this has been fixed for 6 months. Please upgrade your CNI plugins.

And jellonek commented:

This is the question for your deployment software provider, which in your case is probably https://github.com/kubernetes/kubeadm

However, in cases when you need to turn off ipv6 for some reason, you can install a cluster with addon KubeDNS instead of CoreDNS using kubeadm :

kubeadm init --pod-network-cidr=192.168.0.0/16 --feature-gates=CoreDNS=false

Or you can delete CoreDNS deployment and service, and then apply KubeDNS yaml, which should be generated or adjusted to your actual cluster configuration.

Here is a function from the deployAddons.sh file responsible for deploying kube-dns addon:

function deploy_dns {
  echo "Deploying DNS on Kubernetes"
  cp "${KUBE_ROOT}/cluster/addons/dns/kube-dns/kube-dns.yaml.sed" kube-dns.yaml
  sed -i -e "s/\\\$DNS_DOMAIN/${DNS_DOMAIN}/g" kube-dns.yaml
  sed -i -e "s/\\\$DNS_SERVER_IP/${DNS_SERVER_IP}/g" kube-dns.yaml

  KUBEDNS=`eval "${KUBECTL} get services --namespace=kube-system | grep kube-dns | cat"`

  if [ ! "$KUBEDNS" ]; then
    # use kubectl to create kube-dns addon
    ${KUBECTL} --namespace=kube-system create -f kube-dns.yaml

    echo "Kube-dns addon is successfully deployed."
  else
    echo "Kube-dns addon is already deployed. Skipping."
  fi

  echo
}

Answer 2

I have to enable the IPv6 to my CoreDNS pod to run.

Here is the steps I followed:

https://www.thegeekdiary.com/how-to-enable-ipv6-on-centos-rhel-7/

Edit /etc/default/grub and change the value of kernel parameter ipv6.disable from 1 to 0 in line:

# grub2-mkconfig -o /boot/grub2/grub.cfg

# shutdown -r now

Answer 3

I solved the problem without IPv6 enabling (with help from https://www.facebook.com/groups/k8skr )

So, the reason is that, default Kubernetes CNI of Ubuntu is 0.6.0. But the problem is fixed with Kubernetes CNI 0.7.0 So you can update it through download from the below site and replace bin files to /opt/cni/bin/

https://github.com/containernetworking/plugins/releases/tag/v0.7.1

At least, it works for me :)

Answer 4

I fixed this with the reference of these links

check with kubernates troubleshooting guide and the networking plugin addons you have used

https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-or-kube-dns-is-stuck-in-the-pending-state

https://kubernetes.io/docs/concepts/cluster-administration/addons/

And install the following with those and check

kubectl get pods -n kube-system