Why does port-forwarding to a headless service always route to the same pod?

Question

My question is about the Port Forwarding feature applied to a headless service in Kube.netes.

For example this command:

kubectl port-forward service/mongodb-headless 27017:27017 -n mongodb

port-forwards to a headless service. Although I've read that a headless service does neither load balance nor proxy requests to its backing pods, I've observed that it always connects to the same backing pod (selection seems random).

After some investigation I've found out that DNS lookups for a headless service's FQDN returns the IPs of all backing pods in a round-robin based behavior. Regardless re-port-forwarding to my headless service always connects to the same pod. Also a re-deployment of the headless service does not change the selected pod.

Does anyone know about the behavior of port-forwading to a headless service?

• Why does port-forwarding to a headless service bind the pod's port to my localhost port at all?
• Which backing pod (let's say 1 out of 3) is chosen for this binding?
• Why does the headless service always stick to the same pod?

Thanks

Answer 1

Yes, kubectl port-forward always routes you to the same pod, as of November 2022.
This is simply due to how it's implemented under the hood.
It seems to just take the first available endpoint of the service.

Why does port-forwarding to a headless service bind the pod's port to my localhost port at all?

Why not? The original purpose of kubectl port-forward is to target a pod. Then it was improved to also allow users to target a service - but first, it "translates" that service into a suitable pod. This happens to always be the same pod at the moment.

Which backing pod (let's say 1 out of 3) is chosen for this binding?

I suspect it's the first one returned by kubectl get endpoints for that service.

Why does the headless service always stick to the same pod?

Normal services that aren't headless behave exactly the same.
Note that this has nothing to do with the service being headless.

Sources:
https://github.com/kube.netes/kubectl/issues/881 https://github.com/kube.netes/kube.netes/issues/15180#issuecomment-410798267