Kubernetes pod random timeout

Question

I have a Kubernetes deployment containing a very simple Spring Boot web application. I am experiencing random timeouts trying to connect to this application externally.

Some requests return instantly whereas others hang for minutes.

I am unable to see any issues in the logs.

When connecting to the pod directly, I am able to curl the application and get a response immediately so it feels more like a networking issue.

I also have other applications with the identical configuration running in the same cluster which are experiencing no problems.

I am still quite new to Kubernetes so my question would be:

Where and how should I go about diagnosing network issues?

Can provide more information if it helps.

Answer 1

As you have narrow down the issue to networking which means components of the cluster are healthy such as Kubelet, Kube-proxy and etc.

You can check their status by using systemctl utility. For example

systemctl status kubelet
systemctl status kube-proxy

You can get more detail by using journalctl utility. for example

journalctl -xeu kubelet
journalctl -f -u docker

Now If you want to know what's the destiny of the packets then you need to use iptables utility. It's the one who decides forwarding, routing, and verdict of the packets (incoming or outgoing packetes).

My plan of action is Do Not make any assumptions.I follow following utilities to clear the doubts.

• Kubectl

  Kubectl describe pod/svc podName/svcName

• systemctl

• journalctl
• etcdctl
• curl
• iptables

If I still could not solve the issue it means I have made an assumption.

please let me know any other tools I would love to put it on my utility-set