分布式环境中的消息排序

Question

I need some guidance on the solutioning of a problem which I am facing at the moment. I have a microservice which is deployed in 8 pods and it has a JMS listener which is enabled in all the pods to increase parallelism.

Now the event is getting processed in different order and not able to maintain the order of event processing. The business required the events to be processed in a specific order (Event contains a sequence no, which guarantees the order). But we need the parallelism in place to run the listeners in multiple pods.

If anyone have any suggestions of how to design it, please can you share? using DB or redis or any other component?

Thanks Sandeep

Answer 1

Let me take different example, for instance- Kafka, where sequencing is maintain using partition, and all message having same context(ex: user id) will go to same partition. parallelism can be achieve with number of partitions available.

To achieve similar features, parallelism and sequencing of message in JMS is quite tricky. With just one Queue, you can't achieve parallelism and sequencing together. This can be achieve in following ways

• Introduce more queue for same purpose, difference will be just some suffix. Ex: event_consumer_1, event_consumer_2...etc. Queue will be created when consumer initialise if queue already not present.

• Load balance the event message at producer, producer should have all the information about available Queues[we are using in house service discovery mechanism], probably you can explore service discovery framework to share the list of queue available.

• Route the message at producer, using common context/group( user Id, event type, ..depend on your message structure), This will make sure that message of specific type will be sequenced. Like hashing of userId, it will always route the message of specific user Id at same Queue.

• Since all consumer have own queue, this help you to achieve parallelism too

let me know if any specific information require.