Kubernetes - Pod Remains in ContainerCreating Status

Question

I am new to all things Kubernetes so still have much to learn.

Have created a two node Kubernetes cluster and both nodes (master and worker) are ready to do work which is good:

[monkey@k8s-dp1 nginx-test]# kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
k8s-dp1   Ready     master    2h        v1.9.1
k8s-dp2   Ready     <none>    2h        v1.9.1

Also, all Kubernetes Pods look okay:

[monkey@k8s-dp1 nginx-test]# kubectl get pods --all-namespaces
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-dp1                      1/1       Running   0          2h
kube-system   kube-apiserver-k8s-dp1            1/1       Running   0          2h
kube-system   kube-controller-manager-k8s-dp1   1/1       Running   0          2h
kube-system   kube-dns-86cc76f8d-9jh2w          3/3       Running   0          2h
kube-system   kube-proxy-65mtx                  1/1       Running   1          2h
kube-system   kube-proxy-wkkdm                  1/1       Running   0          2h
kube-system   kube-scheduler-k8s-dp1            1/1       Running   0          2h
kube-system   weave-net-6sbbn                   2/2       Running   0          2h
kube-system   weave-net-hdv9b                   2/2       Running   3          2h

However, if I try to create a new deployment in the cluster, the deployment gets created but its pod fails to go into the appropriate RUNNING state. eg

[monkey@k8s-dp1 nginx-test]# kubectl apply -f https://k8s.io/docs/tasks/run-application/deployment.yaml
deployment "nginx-deployment" created

[monkey@k8s-dp1 nginx-test]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                READY     STATUS              RESTARTS   AGE
default       nginx-deployment-569477d6d8-f42pz   0/1       ContainerCreating   0          5s
default       nginx-deployment-569477d6d8-spjqk   0/1       ContainerCreating   0          5s
kube-system   etcd-k8s-dp1                        1/1       Running             0          3h
kube-system   kube-apiserver-k8s-dp1              1/1       Running             0          3h
kube-system   kube-controller-manager-k8s-dp1     1/1       Running             0          3h
kube-system   kube-dns-86cc76f8d-9jh2w            3/3       Running             0          3h
kube-system   kube-proxy-65mtx                    1/1       Running             1          2h
kube-system   kube-proxy-wkkdm                    1/1       Running             0          3h
kube-system   kube-scheduler-k8s-dp1              1/1       Running             0          3h
kube-system   weave-net-6sbbn                     2/2       Running             0          2h
kube-system   weave-net-hdv9b                     2/2       Running             3          2h

I am not sure how to figure out what the problem is but if I for example do a kubectl get ev , I can see the following suspect event:

<invalid>   <invalid>    1         nginx-deployment-569477d6d8-f42pz.15087c66386edf5d   Pod
             Warning   FailedCreatePodSandBox   kubelet, k8s-dp2        Failed create pod sandbox.

But I don't know where to go from here. I can also see that the nginx docker image itself never appears in docker images .

How do I find out more about the problem? Am I missing something fundamental in the kubernetes setup?

--- NEW INFO ---

For background info in case it helps...

Kubernetes nodes are running on CentOS 7 VMs hosted on Windows 10 hyper-v.

--- NEW INFO ---

Running kubectl describe pods shows the following Warning:

Warning  NetworkNotReady         1m                             kubelet, k8s-dp2  network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]

--- NEW INFO ---

Switched off the Hyper-v VMs running Kubernetes for the night after my day job hours were over and on my return to the office this morning, I powered up the Kubernetes VMs once again to carry on and, for about 15 mins, the command:

kubectl get pods --all-namespaces was still showing ContainerCreating for those nginx pods the same as yesterday but, right now, the command is now showing all pods as Running including the nginx pods... ie the problem solved itself after a full reboot of both master and worker node VMs.

I now did another full reboot again and all pods are showing as Running which is good.

Answer 1

使用kubectl describe pod <name>查看更多信息

Answer 2

对运行 Kubernetes 主节点和 Kubernetes 工作节点的两个虚拟机进行完全重启后，Pod 都显示为正在Running （注意：第一次重启后，有问题的 Pod 大约需要 15-20 分钟才能进入Running状态，并且在随后的重新启动时，有问题的 pod 相对更快地进入Running状态...... 3-5 分钟）。

Answer 3

Using kubectl describe pod would show all the events. In some cases, the deployment might be still pulling the docker images from remote, so the status would be still shown as ContainerCreating

Answer 4

尝试systemctl restart kubelet.service

Answer 5

你可以删除de pod，它会自动重新创建。

kubectl delete pod -n namespace podname

Answer 6

I was facing the same issue yesterday. When I describe those pods in ContainerCreating status, the problem was with CNI, it was failing and pods stays in ContainerCreating status. So I delete the CNI from the controlplane and redeploy it. All the pods will change its status within a minute to running status.

Answer 7

就我而言，这是由于在部署命名空间中缺少 Secret 或说 ConfigMap

Answer 8

You can run kubectl describe command on the deployment to be sure as to the events going on or you can run the describe command on the pods that the deployment is spinning up.

Sometimes you may not be having enough resources in your cluster. Check what you using kubectl top command on the pods running to see if one of them is exhausting all of your resources.

I hope this is helpful enough

Answer 9

有同样的问题，但我这边的问题是集群花费太多时间来拉取镜像，快速重启集群可能有助于加快进程

Answer 10

只是分享这个命令对找出我的 ContainerCreating Status 问题有很大帮助：

kubectl get events --sort-by=.metadata.creationTimestamp