如何使用 RabbitMQ 平衡发布者的请求？

Question

Suppose you have multiple producers and one consumer which wants to receive persistent messages from all publishers available.

Producers work with different speed. Let's say that system A produces 10 requests/sec and system B 1 request/sec. So if you use the only queue you will process 10 messages from A then 1 message from B.

But what if you want to balance load and process one message from A then one message from B etc.? Consuming from multiple queues is not a good option because we can't use wildcard binding in this case.

Update:
Queue per producer seems as the best approach. Producers don't know their speed which changes constantly. Having one queue per consumer I can subscribe to one topic and receive messages from all publishers available. But having a queue per producer I need to code the logic by myself:

1. Get all available queues through management plugin (AMQP doesn't allow to list queues).
2. Filter by queue name.
3. Implement round robin strategy.
4. Implement notification mechanism to subscribe to new publishers that can appear at any moment.
5. Remove unnecessary queue when publisher had disappeared and client read all messages.

Well, it seems pretty easy but I thought that broker could provide all of this functionality without any coding. In case with one queue I just create one persistent queue, bind it to a topic exchange then start any number of publishers that send messages to the topic. This option works almost out of the box.

Answer 1

I know I'm late for the party, but still.

In Azure Service Bus terms it's called " partitioning " and it's based on the partition key. The best part is in Azure SB the receiving client is not aware of the partitioning, it simply subscribes to the single queue.

In RabbitMQ there is a X-Consistent-Hashing plugin (" rabbitmq_consistent_hash_exchange ") but unfortunately it's not that convenient. The consumers must be explicitly configured to consume from specific queues. If you have ten queues then you need to setup your consumers so that all ten are covered.

Another two options:

1. Random Exchange Type
2. Sharding Plugin

Bear in mind that with the Sharding Plugin even though it creates "one logical queue to consume" you'll have to have as many subscribers as there are virtual queues, otherwise some of the queues will be left unconsumed.

Answer 2

You can use the Priority Queue Support and associate a priority according to the producer speed. With the caveat that the priority must be set with caution (for example, if the consumer speed is below the system B, the consumer will only consume messages from B) and producers must be aware of their producing speed.

Another option to consider is creating 3 types of queues according to the producing speed: HIGH , MEDIUM , LOW . The three queues are binded to the exchange with the binding key set according to the producing speed. It could be done using.

Consumer will consume messages from these 3 queues using a round robin strategy. With the caveat that producers must be aware of their producing speed.

But the best option may be a queue per producer especially if producers speed is not stable and cannot be categorized. Thus, producers do not need to know their producing speed.