Are all the messages in SQS batch sent to Dead Letter Queue after reaching the maximum redrive(Retry) policy number

Question

I have a SQS queue and the consumer of this queue is a lambda and the unprocessed messages are being sent to a Dead Letter Queue with a redrive policy of 10.

My SQS queue's batch size is 5, will all the messages in a particular batch be sent to the DLQ or only the ones which crossed the retry attempt of 10 and didn't process would go to DLQ.

If 3 messages in a batch of 5 got successfully processed, will all 5 go to DLQ or only the 2 that didn't get processed.

Answer 1

The process of sending msgs to DQL is independent of lambda function or its batch setup. SQS evaluates whether a msg should be send to DLQ on per-message basis. It does not depend on your lambda function or how large batches lambda is using.

So only 2 would be send to DLQ, not entire batch.

Answer 2

The answer is highly dependent and absolutely coupled to what your lambda is actually doing.

How would SQS know whether a message was "successfully processed" or not? Your lambda is iterating over a batch of messages and as long as no error is raised in the handler, the entire batch is considered successful and those messages are acknowledged and removed from the source SQS queue. On the other hand if any error is raised during the iteration through the batch the entire batch is considered a complete failure and all messages would return to the queue.

Post Nov 23, 2021, https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-partial-batch-response-sqs-event-source/ AWS now supports a partial batch response.

Until now, a batch being processed through SQS polling would either be completely successful, in which case the records would be deleted from the SQS queue, or would completely fail, and the records would be kept on the queue to be reprocessed after a 'visibility timeout' period. The Partial Batch Response feature an SQS queue will only retain those records which could not be successfully processed, improving processing performance.

https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting

To avoid reprocessing all messages in a failed batch, you can configure your event source mapping to make only the failed messages visible again. To do this, when configuring your event source mapping, include the value ReportBatchItemFailures in the FunctionResponseTypes list. This lets your function return a partial success, which can help reduce the number of unnecessary retries on records.

Report syntax After you include ReportBatchItemFailures in your event source mapping configuration, you can return a list of the failed message IDs in your function response. For example, suppose you have a batch of five messages, with message IDs id1, id2, id3, id4, and id5. Your function successfully processes id1, id3, and id5. To make messages id2 and id4 visible again in your queue, your response syntax should look like the following:

{ "batchItemFailures": [ { "itemIdentifier": "id2" }, { "itemIdentifier": "id4" } ] }

TLDR;

The default behavior of any exceptions that happen in a lambda is to return the entire batch of messages back to the source. In order to remove only the successfully processed messages and return unsuccessful messages back to the queue, your function must try/catch around each message, save the ids of the unsuccessful messages and return that payload. That will cause the receive count to increment on the unsuccessful messages and then depending on your DLQ policy, once an individual message's receive count is reached that message is diverted to the DLQ.