如何查找 Pod 崩溃的原因？

Question

Is there a way to see why a kubernetes pod is failing with the status "craskLoopBackOff" under a heavy load?

I have a HorizontalPodAutoscaler which never kicks in. In its status it always shows low (Under 50%) cpu and memory usage.

Tailing the application logs within the pods doesnt give any insights either.

Answer 1

Try looking at the Kubernetes events kubectl get events --sort-by='.lastTimestamp'

If you don't get anything meaningful out of events go to the specific node and see the kubelet logs journalctl -u kubelet

Answer 2

To get logs from a pod you should use:

kubectl logs [podname] -p

You can also do kubelet logs but that's mostly for Cluster logs.

If there is no logs that means your application did not produces any logs before the crash. You would need to rewrite the app and for example add a memory dump on crush.

You mentioned that the pod is dying under heavy load but stats shows only 50% utilization. You should login to the pod and check yourself the load, maybe check how many files are being open because maybe you are hitting the limit.

You can read the Kubernetes docs about Application Introspection and Debugging and go over Debugging CrashLoopBackoffs with Init-Containers .

You can also try running your image in Docker and checking logs there. There is a nice documentation about Logs and troubleshooting available.

If you provide more details we might be more helpful.

Answer 3

Below are some obvious reasons for crashloopbackoff, which I have observed:

1. waiting for some condition to be full-filled eg some secrets, failing healthcheck etc
2. pod is running with burstable or besteffort QoS and is getting killed due to non-availability of resources on node

You can run this script to find the possible issues for pods in a namespace: https://github.com/dguyhasnoname/k8s-day2-ops/blob/master/namespace_scripts/debug_app_namespace.sh