Unwanted round robin dispatching

Question

I am struggling to understand the different types of binding (exchanges) in RabbitMQ. Actually I want to implement a certain pattern, and also to understand which exchange/binding/routingkey combination fits this pattern.

I have the followings: a producer which generates 3 types of messages, 3 consumers, one for each kind of message. Producer P generates A, B and C messages. Consumers receive the following messages: ConsumerA only receives A messages, Consumer B only receives B messages, ConsumerC only receives C messages. This worked fine.

Then I added a 4th consumer, D, which is supposed to receive both B and C messages. When a B message is generated, both consumers B and D should receive it. Similarly, when a C message is produced, both C and D consumers should be notified. Instead of that, what actually happens is this: 1. Message A always goes to consumer A only (which is OK). 2. Message B goes alternatively to consumer B and D (once only to B, once only to D, once only to B, once only to D and so on) (wrong) 3. Message C goes alternatively to consumer C and D (once only to C, once only to D, once only to C, once only to D and so on) (wrong)

I did some reading and I understand this is the so-called round-robin style dispatching, but this seems useless to me. I want to be able to propagate the same message to all the consumers that are interested in it. If there are 10 consumers interested in a messages, I don't want only one of them the receive a message and the others 9 of them know nothing about it, I want every time all 10 consumers to receive the same message. Is this doable with a different pattern/exchange type? Am I missing something obvious about how RabbitMQ works?

Here it some relevant code, for anyone interested: http://pastebin.ca/3070838

(A means bonus, B means gaming, C means payment)

Answer 1

This is for Vor, I didn't know how else to post an answer to him without cutting the text.

I quote from the link you gave me: "A message with a routing key set to "quick.orange.rabbit" will be delivered to both queues. Message "lazy.orange.elephant" also will go to both of them. On the other hand "quick.orange.fox" will only go to the first queue, and "lazy.brown.fox" only to the second. "lazy.pink.rabbit" will be delivered to the second queue only once, even though it matches two bindings. "quick.brown.fox" doesn't match any binding so it will be discarded."

The topic exchange sounds wrong to me. The topic exchange looks like the producer of the messages cares or at least knows who should consume each message. This exchange looks like the produces should be aware when new consumers become interested in some messages or existing consumers lose interest in some message, so he can update the routing key accordingly, so the message reach the correct queues. This is not what I want. I want a system in which whatever happens on the consumers' side, it is 100% transparent to the producer. The producer should not need to produce messages with hints to consumers, they should only say "I just produced a message of type A. Exchange, have this message and do whatever you want with it, I don't care". Then the exchange together with other factors (exchange type, bindings, routing keys etc) should decides who gets the message.

OK, I understand that 2 consumers subscribing to the same queue will generate the round robin dispatching, where each consumer would take turns in being notified of one message sent to that queue. That is good to know for the future. But what pattern should I use to have a message reach 2 consumers at the same time while the producer is 100% decoupled from the consumer, as in he should not be aware how many consumers his message should reach?

ps Is it too much if I ask for a "for dummies" explanation of all the 4 exchange types? Maybe I am slower than the average, but I did read the manual and other google found articles, and it is still not clear to me what are the specific of all the exchanges and when they are useful. Maybe if I understand those exchange types, I will figure out myself how to implement the scenario described in the first post.

Answer 2

Problem that you have:

In your case you creating 3 separate queues. When two consumers subscribed to the same queu then RabbitMQ will round robin message between consumers.

How it could be done:

Your setup will have 1 common exchange (topic exchange). Then each worker will create it's own unique queue, then binds it the common exchange with desired routing key(s). This way producer will publish to the same exchange all the time and consumers would get messages they interested in.

Take a look at ropic exchange: https://www.rabbitmq.com/tutorials/tutorial-five-python.html