服务无法与k8s集群中的rabbitmq通信

Question

I have k8s cluster (1.16) with one service in it's specific namespace, and a stateful set of rabbitmq (image: 3.8.0-alpine) with 3 replicas, one on each of my workers, in a different namespace. I configured the service to connect rabbitmq with amqp://user:password@rabbitmq-service.rabbitmq-namespace:5672

I'm getting this error in the logs of my service:

Rabbit mq disconnected connect EHOSTUNREACH <rabbitmq_svc_ip>:5672

rabbitmq seems to be working fine. there are no errors shown in the pods.

What can I check to find the issue and make them communicate?

Answer 1

You asked for how to approach this issue, and so I am including how I would approach it.

I would describe your service to see if your pod ips are listed as endpoints behind it.

kubectl describe service <rabbitmq-service>

If they are not listed, then I would look at your service's selector to make sure that it matches your statefulset. If they are listed, then I would double check that your Service Port is actually 5672, and then I would exec into your pod where your app is running and try to ping the service name in your connection string to make sure it's reachable from your app's namespace. I would also try to do telnet tests to your service name and port to ensure that your app can connect to your service on that port, and I would telnet to your rabbitmq pod ips and ports to ensure that your app can connect to them. If telnet isn't in your image, I would install it for troubleshooting purposes.

kubectl exec -it <app-pod> -n <namespace> /bin/bash

telnet <pod ip or service name> <port>