RabbitMQ Shovel Message Rate

Question

I have a queue with the following parameters:

• x-ha-policy: all
• durable: true

There is a static shovel configured to process this queue and another server is set as destination. The shovel configuration:

{sources, [ {broker, "amqp://web:web@thisserver.spc/%2F"} ]},
{destinations, [ {broker, "amqp://web:web@remoteserver.spc/%2F"} ]},
{queue, <<"queuename">>},
{prefetch_count, 10},
{publish_fields, [ {exchange, <<"exchangename">>} ]},
{reconnect_delay, 5}

My problem is that the deliver / get and ack rates never go over 50/s for the "queuename" queue, therefore causing a huge build-up (7 million) in the queue.

I wonder if changing the prefetch_count will increase the message rate?

Also "Ack required" is enabled for the consumer of the queue. The default value for this is: on_confirm. If the messages are forced to be acknowledged, does it limit the message rates?

Answer 1

There could be a few things going on here that could affect the message rates including network speed, message size, memory size for the RabbitMQ Broker, etc. You can change the pre-fetch count and make it higher, that will help some. Due to the high number of messages in the queue you may be reaching flow control, check out this article .