Kubernetes pod on Google Container Engine continually restarts, is never ready

Question

I'm trying to get a ghost blog deployed on GKE, working off of the persistent disks with WordPress tutorial . I have a working container that runs fine manually on a GKE node:

docker run -d --name my-ghost-blog -p 2368:2368 -d us.gcr.io/my_project_id/my-ghost-blog

I can also correctly create a pod using the following method from another tutorial:

kubectl run ghost --image=us.gcr.io/my_project_id/my-ghost-blog --port=2368

When I do that I can curl the blog on the internal IP from within the cluster, and get the following output from kubectl get pod :

Name:       ghosty-nqgt0
Namespace:      default
Image(s):     us.gcr.io/my_project_id/my-ghost-blog
Node:       very-long-node-name/10.240.51.18
Labels:       run=ghost
Status:       Running
Reason:
Message:
IP:       10.216.0.9
Replication Controllers:  ghost (1/1 replicas created)
Containers:
  ghosty:
    Image:  us.gcr.io/my_project_id/my-ghost-blog
    Limits:
      cpu:    100m
    State:    Running
      Started:    Fri, 04 Sep 2015 12:18:44 -0400
    Ready:    True
    Restart Count:  0
Conditions:
  Type    Status
  Ready   True
Events:
  ...

The problem arises when I instead try to create the pod from a yaml file, per the Wordpress tutorial. Here's the yaml:

metadata:
  name: ghost
  labels:
    name: ghost
spec:
  containers:
    - image: us.gcr.io/my_project_id/my-ghost-blog
      name: ghost
      env:
        - name: NODE_ENV
          value: production
        - name: VIRTUAL_HOST
          value: myghostblog.com
      ports:
        - containerPort: 2368

When I run kubectl create -f ghost.yaml , the pod is created, but is never ready:

> kubectl get pod ghost
NAME      READY     STATUS    RESTARTS   AGE
ghost     0/1       Running   11         3m

The pod continuously restarts, as confirmed by the output of kubectl describe pod ghost :

Name:       ghost
Namespace:      default
Image(s):     us.gcr.io/my_project_id/my-ghost-blog
Node:       very-long-node-name/10.240.51.18
Labels:       name=ghost
Status:       Running
Reason:
Message:
IP:       10.216.0.12
Replication Controllers:  <none>
Containers:
  ghost:
    Image:  us.gcr.io/my_project_id/my-ghost-blog
    Limits:
      cpu:    100m
    State:    Running
      Started:    Fri, 04 Sep 2015 14:08:20 -0400
    Ready:    False
    Restart Count:  10
Conditions:
  Type    Status
  Ready   False
Events:
  FirstSeen       LastSeen      Count From              SubobjectPath       Reason    Message
  Fri, 04 Sep 2015 14:03:20 -0400 Fri, 04 Sep 2015 14:03:20 -0400 1 {scheduler }                      scheduled Successfully assigned ghost to very-long-node-name
  Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} implicitly required container POD created   Created with docker id dbbc27b4d280
  Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} implicitly required container POD started   Started with docker id dbbc27b4d280
  Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} spec.containers{ghost}      created   Created with docker id ceb14ba72929
  Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} spec.containers{ghost}      started   Started with docker id ceb14ba72929
  Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} implicitly required container POD pulled    Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
  Fri, 04 Sep 2015 14:03:30 -0400 Fri, 04 Sep 2015 14:03:30 -0400 1 {kubelet very-long-node-name} spec.containers{ghost}      started   Started with docker id 0b8957fe9b61
  Fri, 04 Sep 2015 14:03:30 -0400 Fri, 04 Sep 2015 14:03:30 -0400 1 {kubelet very-long-node-name} spec.containers{ghost}      created   Created with docker id 0b8957fe9b61
  Fri, 04 Sep 2015 14:03:40 -0400 Fri, 04 Sep 2015 14:03:40 -0400 1 {kubelet very-long-node-name} spec.containers{ghost}      created   Created with docker id edaf0df38c01
  Fri, 04 Sep 2015 14:03:40 -0400 Fri, 04 Sep 2015 14:03:40 -0400 1 {kubelet very-long-node-name} spec.containers{ghost}      started   Started with docker id edaf0df38c01
  Fri, 04 Sep 2015 14:03:50 -0400 Fri, 04 Sep 2015 14:03:50 -0400 1 {kubelet very-long-node-name} spec.containers{ghost}      started   Started with docker id d33f5e5a9637
...

This cycle of created/started goes on forever, if I don't kill the pod. The only difference from the successful pod is the lack of a replication controller. I don't expect this is the problem because the tutorial mentions nothing about rc.

Why is this happening? How can I create a successful pod from config file? And where would I find more verbose logs about what is going on?

Answer 1

If the same docker image is working via kubectl run but not working in a pod, then something is wrong with the pod spec. Compare the full output of the pod as created from spec and as created by rc to see what differs by running kubectl get pods <name> -o yaml for both. Shot in the dark: is it possible the env vars specified in the pod spec are causing it to crash on startup?

Answer 2

Maybe you could use different restart Policy in the yaml file?

What you have I believe is equivalent to

- restartPolicy: Never

no replication controller. You may try to add this line to yaml and set it to Always (and this will provide you with RC), or to OnFailure.

https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/pod-states.md#restartpolicy

Answer 3

Container logs may be useful, with kubectl logs

Usage:

kubectl logs [-p] POD [-c CONTAINER]

http://kubernetes.io/v1.0/docs/user-guide/kubectl/kubectl_logs.html