GCP GKE：查看终止的作业/窗格的日志

Question

I have a few cron jobs on GKE.

One of the pods did terminate and now I am trying to access the logs.

➣ $ kubectl get events
LAST SEEN   TYPE      REASON             KIND      MESSAGE
23m         Normal    SuccessfulCreate   Job       Created pod: virulent-angelfish-cronjob-netsuite-proservices-15622200008gc42
22m         Normal    SuccessfulDelete   Job       Deleted pod: virulent-angelfish-cronjob-netsuite-proservices-15622200008gc42
22m         Warning   DeadlineExceeded   Job       Job was active longer than specified deadline
23m         Normal    Scheduled          Pod       Successfully assigned default/virulent-angelfish-cronjob-netsuite-proservices-15622200008gc42 to staging-cluster-default-pool-4b4827bf-rpnl
23m         Normal    Pulling            Pod       pulling image "gcr.io/my-repo/myimage:v8"
23m         Normal    Pulled             Pod       Successfully pulled image "gcr.io/my-repo/my-image:v8"
23m         Normal    Created            Pod       Created container
23m         Normal    Started            Pod       Started container
22m         Normal    Killing            Pod       Killing container with id docker://virulent-angelfish-cronjob:Need to kill Pod
23m         Normal    SuccessfulCreate   CronJob   Created job virulent-angelfish-cronjob-netsuite-proservices-1562220000
22m         Normal    SawCompletedJob    CronJob   Saw completed job: virulent-angelfish-cronjob-netsuite-proservices-1562220000

So at least one CJ run.

I would like to see the pod's logs, but there is nothing there

➣ $ kubectl get pods
No resources found.

Given that in my cj definition, I have:

failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 3

shouldn't at least one pod be there for me to do forensics?

Answer 1

You can use the --previous flag to get the logs for the previous pod.

So, you can use:

kubectl logs --previous virulent-angelfish-cronjob-netsuite-proservices-15622200008gc42

to get the logs for the pod that was there before this one.

Answer 2

Your pod is crashing or otherwise unhealthy

First, take a look at the logs of the current container:

kubectl logs ${POD_NAME} ${CONTAINER_NAME}

If your container has previously crashed, you can access the previous container's crash log with:

kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}

Alternately, you can run commands inside that container with exec:

kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN}

Note: -c ${CONTAINER_NAME} is optional. You can omit it for pods that only contain a single container.

As an example, to look at the logs from a running Cassandra pod, you might run:

kubectl exec cassandra -- cat /var/log/cassandra/system.log

If none of these approaches work, you can find the host machine that the pod is running on and SSH into that host.

Finaly, check Logging on Google StackDriver.

Debugging Pods

The first step in debugging a pod is taking a look at it. Check the current state of the pod and recent events with the following command:

kubectl describe pods ${POD_NAME}

Look at the state of the containers in the pod. Are they all Running? Have there been recent restarts?

Continue debugging depending on the state of the pods.

Debugging ReplicationControllers

ReplicationControllers are fairly straightforward. They can either create pods or they can't. If they can't create pods, then please refer to the instructions above to debug your pods.

You can also use kubectl describe rc ${CONTROLLER_NAME} to inspect events related to the replication controller.

Hope it helps you to find exactly problem.