Multiple Consumers with Amazon SQS

Question

I code all my micro-service in java. I want use Multiple Consumers with Amazon SQS but each consumer has multiple instances on AWS behind a load balancer.

I use SNS for input stream

I use SQS Standard Queue after SNS.

I find the same question on stackoverflow ( Using Amazon SQS with multiple consumers )

This sample is

https://aws.amazon.com/fr/blogs/aws/queues-and-notifications-now-best-friends/

When I read SQS Standard Queue documentation, I see that occasionally more than one copy of a message is delivered. :

Each message has a message_id . How to detect that there are not multiple instances of a same micro-service processes the same message that would have been sent multiple times. I got an idea by registering the message_id in a dynamodb database but if this is done by several instances of the same micro-service, how to make a lock on the get (a bit like a SELECT FOR UPDATE)?

for example multiple instances of a same micro-service "Scan Metadata".

Answer 1

As you have mentioned, standard SQS queues can sometimes deliver the same message more than once. This is due to the distribute nature of SQS service. Each message is stored on multiple servers for redundancy and there is a change that one of those servers is down when you are calling sqs:DeleteMessage , therefore the message will not be deleted from all of the servers and once the failed server comes back online, it doesn't know that the you have deleted the message and it will be processed again.

Easiest way to solve the issue with duplicate messages is to switch to using FIFO queue which provides you with exactly once processing. You can choose to use deduplication based on either content or unique ID generated by sender. If you choose to use content deduplication, when queue receives two messages with the same content in 5 min. deduplication interval, the message will be discarded.

If two messages can have the same content yet you need to treat them as different messages, you can use deduplication based on ID that you can pass to sqs:SendMessage or sqs:SendMessageBatch calls via MessageDeduplicationId argument.

I would definitely check FIFO queues before thinking about using DynamoDB to store the state of message processing. It will be cheaper and this deduplication functionality is provided for you by default without you having to implement any complex logic.