In pub-sub pattern can I track msg completion

Question

I have two microservices and they are talking to each other using pub sub pattern.

I want my serviceA to publish 100 messages of certain type and once all the messages are processed successfully then only I want next 100 messages to get published.

Does anyone have any idea how can track the status of each message and to let my service know that first 100 are completed, now publish next 100?

Answer 1

The pub-sub pattern is in unidirectional. The publisher/producer send information/command to its subscribers/consumers.

So, the pattern itself does not support bidirectional communication. But you can use the same mechanism to send information/command backward. With two channels (one request, one reply) you can have a bidirectional protocol.

In your particular case you have two communicating parties so both channels would have a single subscriber.

It is a good practice to include a requestId in the request and reply messages. With that you could correlate them. In your case this could mean a batchId. The 100 request messages include the same batchId. The single reply message tells that a given batch has been successfully processed.

Answer 2

I suggest you look into Sagas or Workflows. Both a Saga as well as a Workflow implement a process, that is, a sequence of steps, often called activities.

When using Sagas (Example: Sagas with NServiceBus ), you are responsible to track the state of your process, for example, you have to track the list of requests to be sent and the list of responses to be received before moving on manually.

"Looks like a state machine" ™️

When using Workflows (Example: temporal.io ), the workflow is responsible to track the state of your process, for example, you can send all requests and await all responses before moving on automatically.

"Looks like a function" ™️

(Disclaimer I work at temporal.io )