在背压的情况下，AWS SQS 块出队

Question

In case of lambda workers processing batches from an SQS queue, Is there an option to monitor the worker's failure rate (wrt processing job) and block further dequeueing (and as a result, lambda invocations) in case failure rate crosses a threshold? I can monitor lambda's error/invocation rate, but how would the dequeue halting be implemented? I don't want to empty the queue and lose the data.

Answer 1

First thing is to understand why your Lambda could (possibly) be failing:

1) If they are failing because of throttling (more messages to be processed than available Lambda functions), the message (or the whole batch) will be sent back to the Queue and will be tried again once the Visibility Timeout expires, so the retry logic is already built-in for you and scales well.

2) If they are failing because of bad messages or some error in the code, you can configure a DLQ to send the failed messages to. This is easy to setup as you only need to tell your Lambda function which DLQ to connect to in case of failure.

If you scenario is 1), rest assured your messages won't be lost. If your scenario is 2), just configure a DLQ for further analysis of the failed messages.

You can also check the official docs to understand Lambda's Retry Behaviour