RabbitMQ (Java) multiple consumers performance issue

Question

I'm implementing a daily job which get data from a MongoDB (around 300K documents) and for each of them publish a message on a RabbitMQ queue. On the other side I have some consumers on the same queue, which ideally should work in parallel.

Everything is working but not as much as I would, specially regarding consumers performances.

This is how I declare the queue:

rabbitMQ.getChannel().queueDeclare(QUEUE_NAME, true, false, false, null);

This is how the publishing is done:

rabbitMQ.getChannel().basicPublish("", QUEUE_NAME, null, body.getBytes());

So the channel used to declare the queue is used to publish all the messages.

And this is how the consumers are instantiated in a for loop (10 in total, but it can be any number):

Channel channel = rabbitMQ.getConnection().createChannel();
MyConsumer consumer = new MyConsumer(customMapper, channel, subscriptionUpdater);
channel.basicQos(1);    // also tried with 0, 10, 100, ...
channel.basicConsume(QUEUE_NAME, false, consumer);

So for each consumer I create a new channel and this is confirmed by logs:

...
com.rabbitmq.client.impl.recovery.AutorecoveringChannel@bdd2027
com.rabbitmq.client.impl.recovery.AutorecoveringChannel@5d1b9c3d
com.rabbitmq.client.impl.recovery.AutorecoveringChannel@49a26d19
...

As far as I've understood from my very short RabbitMQ experience, this should guarantee that all the consumer are called. By the way, consumers need between 0.5 to 1.2 seconds to complete their task. I have just spotted very few 3 seconds.

I have two separate queues and I repeat what I said above two times (using the same RabbitMQ connection).

So, I have tested publishing 100 messages for each queue. Both of them have 10 consumers with qos=1.

I didn't expect to have exactly a delivery/consume performance of 10/s, instead I noticed:

• actual values are around 0.4 and 1.0.
• at least all the consumers bound to the queue have received a message, but it doesn't look like "fair dispatching".
• it took about 3 mins 30 secs to consume all the messages on both queues.

Am I missing the main concept of threading within RabbitMQ? Or any specific configuration which might be still at default value? I'm on it since very few days so this might be possible.

Please notice that I'm in the fortunate position where I can control both publishing and consuming parts :)

I'm using RabbitMQ 3.7.3 locally, so it cannot be any network latency issue.

Thanks for your help!

Answer 1

The setup of RabbitMQ channels and consumers were correct in the end: so one channel for each consumer.

The problem was having the consumers calling a synchronized method to find and update a MongoDB document.

This was delaying the execution time of some consumers: even worst, the more consumers I was adding (thinking to speed up processing), the less message rate/s I was getting.

I have moved the MongoDB part on he publishing side where I don't have to care about synchronization because it's done in sequence by one publisher only. I have a slightly decreased delivery rate/s but now with just 5 consumers I easily reach an ack rate of 50-60/s.

Lessons learnt:

• create a separate channel for the publisher.
• create a separate channel for each consumer.
• let RabbitMQ manage threading for the consumers (--> you can instantiate them on the main thread).
• (if possible) back off publishing to give the queues 100% time to deal with consumers.
• set a qos > 1 for each consumer channel. But this really depends on your scenario and architecture: you must do some performance test.

As a general rule:

• (1) calculate/estimate delivery time.
• (2) calculate/estimate ack time.
• (3) calculate/estimate consumer time.
• qos = (1) + (2) + (3) / (3)

This will give you an initial qos value to test and tweak based on your scenario. The final goal is to have 100% utilization for all the available consumers.