How to process messages from SQS queues by groups

Question

I currently have a SQS queue that triggers a lambda function, but SQS have a 120k limit on in-flight messages, meaning that only 120k messages can be processed by the lambda consumer. That works fine for the most case. But imagine I have clients A and B, if A sends 500k to the queue and after that B sends only 1 message, B needs to wait all messages in the queue to be processed. It does not make sense to create one queue for each client (at least not manually). How can I, for example, process messages from clients in a round-robin manner, give everyone the same time to process their tasks?

I have been looking into Kinesis data stream, but not so sure if this will solve my problem. Would I need to create an SNS that redirects to a lambda responsible to round-robin the message to queues that do the same thing but this way I would avoid sending client B to same queue as client A?

Answer 1

You can check out SQS Fifo Queues. Fifo Queues have a concept of grouping using the group Id attribute. all messages in one group are processed in order. The limitations are you can only process one message per group at any given moment.

The above concept is the same as having one queue per tenant, just using one dedicated Queue.

Refer to the AWS Docs

https://aws.amazon.com/blogs/compute/solving-complex-ordering-challenges-with-amazon-sqs-fifo-queues/

Article on ordering inside SQS

https://aws.amazon.com/blogs/compute/solving-complex-ordering-challenges-with-amazon-sqs-fifo-queues/

Note:

FIFO queues are more expensive than standard queues, please refer to the pricing guide before making any decisions.