RabbitMQ several consumers in the same process

Question

I have a .NET project that needs to read messaged from a given Queue. I have several producers writing the same type of message into the queue. I want my consumer app to have several threads reading messages and handling them so that the load will not be on a single thread.

Any ideas or sample code on how to achieve this? Again, Note: Each message should be processed once and not several times. The work should be balanced between the worker threads

Answer 1

You are going to need a bit of plumbing to get that done.

I have an open-source service bus called Shuttle.Esb and there are many other service bus options available that you may wish to consider.

But if you do not want to go that route you could still have a look at some of the code and implementations to get some ideas. I have a RabbitMQ implementation that may be of assistance.

Answer 2

Take a look at masstransit project : http://masstransit-project.com/MassTransit/usage/message-consumers.html

It has configurations like prefetch count and concurrency limit. It brings you to consume messages paralelly.

Also it is very simple to setup:

IBusControl busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
    IRabbitMqHost host = cfg.Host(new Uri(RabbitMQConstants.RabbitMQUri),
        hst =>
        {
            hst.Username(RabbitMQConstants.RabbitMQUserName);
            hst.Password(RabbitMQConstants.RabbitMQPassword);
        });

    cfg.ReceiveEndpoint(host,
        RabbitMQConstants.YourQueueName,
        endPointConfigurator => { 
            endPointConfigurator.Consumer<SomeConsumer>();
            endPointConfigurator.UseConcurrencyLimit(4);
        });
});

busControl.Start();

public class SomeConsumer :
    IConsumer<YourMessageClass>
{
    public async Task Consume(ConsumeContext<YourMessageClass> context)
    {
        await Console.Out.WriteLineAsync($"Message consumed: {context.Message.YourValue}");

    }
}