Rebus: 2 handlers in 2 processes. Hit inconsistently and alternately

Question

I have two console apps using Rebus. They both reference an assembly where messages (commands and events) are defined. Console app "A" sends commands and listens to events for logging purposes (eg: sends a CreateTCommand and listens for TCreatedEvent). Console app "B" (which is actually an ASP.NET Core app) listens to commands and handles them (eg: a saga is initiated by CreateTCommand, the aggregate is created and it raises a TCreatedEvent). In another DLL inside the process of app "B" there is a handler for TCreatedEvent.

So I have a creation command sent by app "A" and two handlers for the created event, one in app "A" and one in app "B".

The problem: when I send a command from app "A" the first time, app "B" raises the created event which triggers the handler in the same process. The handler in app "A" is not triggered. Further commands from app "A" are always handled by the saga in app "B" but created events never again hit the handler in that process, but are handled by app "A"!!! Sometimes (I can't understand how to reproduce) commands from app "A" are not handled by the saga in app "B" (I find the command in the error queue in MSMQ with the exception "message with Id could not be dispatched to any handlers"). Sometimes (very rarely) both handlers have been hit. But I can't reproduce the behavior consistently...

My sensations about this (knowing nearly nothing about Rebus, which is quite new to me):

• could it be a concurrency problem? I mean: Rebus is configured to store subscriptions externally to the processes (using SQL or Mongo, the problem doesn't go away), so I thought that maybe the first handler is too fast and marks the event as handled before the second handler is invoked
• checking the subscriptions SQL table, I find 5 rows (one per type of Event that I've subscribed with in the code (using bus.Subscribe() in the app startup) with the same address (the queue name chained to my local machine name). Is it a problem to have only one address with 2 processes trying to consume it?

The configuration code for Rebus is the same in the 2 apps and goes like this:

        const string inputQueueAddress = "myappqueue";
        var mongoClient = new MongoClient("mongodb://localhost:27017");
        var mongoDatabase = mongoClient.GetDatabase("MyAppRebusPersistence");
        var config = Rebus.Config.Configure.With(new NetCoreServiceCollectionContainerAdapter(services))
            .Logging(l => l.Trace())
            .Routing(r => r.TypeBased()
                .MapAssemblyOf<AlertsCommandStackAssemblyMarker>(inputQueueAddress)
                .MapAssemblyOf<AlertsQueryStackAssemblyMarker>(inputQueueAddress)
            )
            .Subscriptions(s => s.StoreInMongoDb(mongoDatabase, "subscriptions"))
            .Sagas(s => s.StoreInMongoDb(mongoDatabase))
            .Timeouts(t => t.StoreInMongoDb(mongoDatabase, "timeouts"))
            .Transport(t => t.UseMsmq(inputQueueAddress));

        var bus = config.Start();
        bus.Subscribe<AlertDefinitionCreatedEvent>();
        bus.Subscribe<AlertStateAddedEvent>();
        bus.Subscribe<AlertConfigurationForEhrDefinitionAddedEvent>();
        services.AddSingleton(bus);

        services.AutoRegisterHandlersFromThisAssembly();

I hope someone can help, this is driving me nuts...

ps: the problem is present also when passing isCentralized: true to subscription.StoreInMongoDb().

EDIT 1: I added console logging and you can see this strange behavior: https://postimg.org/image/czz5lchp9/

The first command is sent successfully. It is handled by the saga and the event triggered the handler in console app "A". Rebus says the second command was not dispatched to any handlers but it was actually handled by the saga (I followed the code in debug) and the event was handled by the handler in app "B" and not "A"... why? ;(

EDIT 2: I'm debugging Rebus's source code. I noticed that in ThreadPoolWorker.cs, method TryAsyncReceive

    async void TryAsyncReceive(CancellationToken token, IDisposable parallelOperation)
    {
        try
        {
            using (parallelOperation)
            using (var context = new DefaultTransactionContext())
            {
                var transportMessage = await ReceiveTransportMessage(token, context);

                if (transportMessage == null)
                {
                    context.Dispose();

                    // no need for another thread to rush in and discover that there is no message
                    //parallelOperation.Dispose();

                    _backoffStrategy.WaitNoMessage();
                    return;
                }

                _backoffStrategy.Reset();

                await ProcessMessage(context, transportMessage);
            }
        }

after the TCreatedEvent is published by app "B", in app "A" the code reaches await ProcessMessage(context, transportMessage) where transportMessage is the actual event. This line of code is not reached in app "B"'s process. Seems like the first receiver of the message removes it from MSMQ's queue. As I said I'm quite new to Rebus and buses in general, but if this behavior is as designed I'm quite puzzled... how can multiple buses in multiple processes listen to the same queue???

Answer 1

You should never have two bus instances receiving messages from the same queue, unless they are multiple instances of the same endpoint .

When two processes use the same input queue, they will take messages from each other. This can be absolutely fine if you are implementing the competing consumers pattern , using it to distribute work evenly between a cluster of worker processes, but you cannot share an input queue between multiple different endpoints.

My guess is that everything will seem much more predictable if you let each bus instance use its own input queue ;)

Now you're telling me these handlers need to live inside the main app

No I'm not :) I'm telling you that you will get unpredictable results if you let two different applications snatch each other's messages.

While it's perfectly fine for all handlers to be in the same bus instance (and thus be invoked by messages from the same queue), the most common scenario is that you will split your application in a way that matches how you want to evolve the system.

This way you can update a single application at a time, avoiding big "stop the world"-updates.

You do this by starting several endpoints, each using its own queue – and then you ROUTE messages between endpoints in order to communicate.

Consider a scenario where you want to send a command to a command processor. The command processor is a Rebus endpoint that gets its messages from the command_processor queue.

In the sender's end you will configure an "endpoint mapping" (you can read more about that in the routing section on the Rebus wiki which could look like this:

Configure.With(...)
    .Transport(t => t.UseMsmq("sender"))
    .Routing(r => {
        r.TypeBased()
            .Map<TheCommand>("command_processor");
    })
    .Start();

which will enable the sender to simply go

await bus.Send(new TheCommand(...));

and then the bus will know which queue to deliver the command message to.

I hope that makes it more clear :)

Please note that this is a very simple case of point-to-point messaging where one endpoint sends a message which is meant to be consumed by one single other endpoint. There exists several other patterns that Rebus can help you with, eg request/reply and publish/subscribe .