Can a NATS publisher send a single message into multiple queues?

Question

I'm building a system where two different entities need to process messages from the same source (in different ways - for example one will log all messages while another entity might want to aggregate data).

Ideally each entity is fully scalable for performance and resiliency, so we have multiple publishers, multiple log subscribers and multiple aggregation subscribers, but still each messages generated by each published is processed by exactly one log subscriber and one aggregation subscribers.

With AMQP we can achieve this by publishing to a fan-out exchange that distributes messages to two queues where each queue has many subscribers. I understand that the same behavior can be achieved in NATS by simply having all the subscribers listen on the same "subject" by use two distinct "queue group names" base on their roles.

In such an instance messages to the subject will be delivered to one subscriber from each queue group, ie each message will be delivered exactly n-times, n being the number of distinct queue groups and not the number of subscribers. Is this correct?

Answer 1

Indeed, you can use queue subscribers (for instance in Go, it would be such API: func (nc *Conn) QueueSubscribe(subj, queue string, cb MsgHandler) (*Subscription, error) .

The queue is the group name. For instance, it could be in your example log and aggregation . You can create as many queue subscribers on each of these groups and only 1 member in each group will receive a given message.

For instance, suppose that you publish a message on subject foo and you have 10 queue subscribers on foo with queue name log and 10 queue subscribers on foo with queue name aggregation . The message will be delivered to 2 subscribers, 1 for group log and 1 for group aggregation .

Hope this helps.

Answer 2

Your approach is correct, the concept of queue in nats.io is to sequential distribute the message among the subscribers listening to the queue. This distribution happens in a linera fashion, suppose you have 10 subscribers (S1- S10) listening to a topic and registered on same queue, then the first message will be sent to S1, then to S2 and so on in a cyclic manner.

You just need to make sure that all the subscribers are connected to the server as if a suscriber goes offline, the nats server will become aware of this event after certain outstanding PING-PONG requests and during that interval it will be forwarding the messages to the offline node. Thus you need to carefully set

1. PING-PONG interval
2. Max outstanding PING requests

https://nats.io/documentation/server/gnatsd-config/