Reserved concurrency on aws lambda does not prevent lambda to scale more?

Question

I have daily scheduled task that triggers around 10k lambda functions for 10k records that I need to maintain. I'm using SQS to queue all those messages and I want to spread execution over couple of hours. So I set up reserved concurrency to only 3 concurrent invocations. But still when that scheduled task hits concurrent invocations of that lambda functions goes over 3. Any advice on how to do it? When I check lambda configuration it shows that reserved concurrency is 3. But on monitoring concurrent invocations shows way over 3.

Answer 1

It's always tricky to use SQS with Lambda (concurrency limit configured) because in short, is not gonna work, instead, you will get some throttling records because the lambda can't process messages limited by the concurrency.

You can check this article which explains the why and a workaround solution: https://zaccharles.medium.com/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0

check also this AWS documentation for further information about this subject https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#events-sqs-queueconfig

Answer 2

In early 2023, AWS Lambda introduced support for setting Maximum Concurrency on the Amazon SQS event source, a more direct and less fiddly way to control concurrency than with reserved concurrency.

The Maximum Concurrency setting limits the number of concurrent instances of the function that an Amazon SQS event source can invoke. The valid range is 2-1000 concurrent instances.

The Lambda Event Source Mapping create and update APIs now have a ScalingConfig option for SQS:

aws lambda update-event-source-mapping \
    --uuid "a1b2c3d4-5678-90ab-cdef-11111EXAMPLE" \
    --scaling-config '{"MaximumConcurrency":3}' # valid range is 2-1000