Kubernetes Pod to Pod communication

Question

I have 2 Deployment - A(1 replica) and B(4 replica)

I have scheduled job in the POD A and on successful completion it hits endpoint present in one of the Pods from Deployment B through the service for Deployment B.

Is there a way I can hit all the endpoints of 4 PODS from Deployment B on successful completion of job?

Ideally one of the pod is notified!! But is this possible as I don't want to use pub-sub for this.

Answer 1

Is there a way I can hit all the endpoints of 4 PODS from Deployment B on successful completion of job?

But is this possible as I don't want to use pub-sub for this.

As you say, a pub-sub solution is best for this problem. But you don't want to use it.

Use stable network identity for Service B

To solve this without pub-sub, you need a stable network-identity for the pods in Deployment B . To get this, you need to change to a StatefulSet for your service B.

StatefulSets are valuable for applications that require one or more of the following.

• Stable, unique network identifiers.

When B is deployed with a StatefulSet, your job or other applications can reach your pods of B, with a stable network identity that is the same for every version of service B that you deploy. Remember that you also need to deploy a Headless Service for your pods.

Scatter pattern: You can have an application aware (eg aware of number of pods of Service B ) proxy, possibly as a sidecar. Your job sends the request to this proxy. The proxy then sends a request to all your replicas. As described in Designing Distributed Systems: Patterns and Paradigms

Pub-Sub or Request-Reply

If using pub-sub , the job only publish an event. Each pod in B is responsible to subscribe.

In a request-reply solution, the job or a proxy is responsible for watching what pods exists (unless it is a fixed number of pods) in service B, in addition it need to send request to all, if requests fails to any pod (it will happen on deployments sometime) it is responsibly to retry the request to those pods.

So, yes, it is a much more complicated problem in a request-reply way.

Answer 2

Kubernetes service is an abstraction to provide service discovery and load balancing. So if you are using service your request will be sent to one of the backend pods.

To achieve what you want I suggest you create 4 different services each having only one backend pod or use a message queue such as rabbitmq between service A and service B.

Answer 3

You can use headless service . That way Kubernetes won't allocate separate IP address and instead will setup DNS record with IP addresses of all the pods. Then in your application just resolve the record and send notification to all endpoints. But really, this is ideal use case for pub-sub or service discovery system. DNS is too unreliable for this.

Answer 4

PUB-SUB is the best option here. I have similar use case and I am using pub-sub , which is in production for last 6 months.