RabbitMQ 多线程多消费者负载均衡

Question

I'm fairly new to RabbitMQ and I've been tasked with fixing a load balancing bug that is occurring.

Here is the setup: We operate a Protobuf over RPC Api utilizing RabbitMQ.

A request to the api will be published to rabbit which will be then be consumed by one of 3 instances (the API is distributed across 3 instances of the application).

Each instance is set to utilize 10 threads to process the request and return a response. We accomplish this by setting a up a SimpleMessageListener with 10 ConcurrentConsumers which results in 10 corresponding threads that handles each message.

This results in:

Instance 1: Consumers 1-10, Instance 2: Consumers 11-20, Instance 3: Consumers 21-30. 

Since Rabbit uses a round robin approach to distribute load across the consumers (vs distributing the load across the instances of the application) if 5 messages come through we end up seeing load like this:

Instance 1: Threads 1-5 in use (6-10 idle), Instance 2: Idle, Instance 3: idle. 

What I want to have happen would be: 5 Messages:

Instance 1: Threads 1,2 in use, Instance 2: threads 1,2 in use Instance 3: Thread 1 in use. 

In other words I would like to go round robin according to the instance (ex. Instance 1(consumer 1), 2(11), 3(21), 1(2), 2(12)) and not the consumer (ex. Instance 1, consumers 1-5).

Is this possible using Spring AMQP (1.2.1) and Spring Rabbit (1.2.1) libraries? My first thought is to reduce the concurrent consumers per instance down to 1 (so rabbit distributes evenly to each instance of the application) and then use an executor to spin up a thread for each request (up to 10 threads total per instance).

I haven't seen anyone else facing a similar issue so I'm hoping someone can give me some guidance! I'm mainly trying to see if I can do config changes to our current set up or if I'll need to implement something manually.

Thanks!

Answer 1

If you want to round-robin to your three servers, then I think that your thought (concurrent consumers = 1) to be the best option.

You should note though, that it's only as fair as the order in which the applications connect to listen for messages, for example:

1. app1 listens
2. app2 listens
3. message dispatched to app1
4. app1 listens
5. app3 listens
6. message dispatched to app2
7. message dispatched to app1 *** this is not round-robin here
8. message dispatched to app3

Without having some kind of targeting information inside your messages, your best effort is going to be as orderly as your processes connect to receive messages.

But if you go down to a consumer count of 1 and hand off the message to an executor and immediately listen for more messages, I would wager you would get a pretty even distribution.

I should also note, you're going to make ACK/NACK harder in this setup. If you're using Java 8, the CompletableFuture API would fit nicely here to do some straightforward ACK/NACK from the handed off threads

Answer 2

You can achive this behaviour with consumer priorization . Just give every consumer on each instance a priorization from 1-10. Since documentation this should spread work load evenly.