AWS Lambda 和 SQS：故障报告

Question

It's more of a "best practice" or "am I doing it right" question, but I can't find the answer anywhere on the Internet. I'm preparing an AWS Lambda (Java), which will be triggered by SQS events. It will receive up to 10 events and process them. If some of the events will not be processed correctly (dependent on external services), it will remove processed events from the SQS queue. Nothing very fancy really:) Question is - how should I finish the execution of the lambda in case of failure in the processing of the batch (either partial or full).

Code for Lambda handler looks like this:

public class LambdaHandler implements RequestHandler<SQSEvent, Void> {

    private final CopierComponent copierComponent;

    public LambdaHandler() {
        this.copierComponent = DaggerCopierComponent.builder().build();
    }

    @Override
    public Void handleRequest(SQSEvent sqsEvent, Context context) {
        context.getLogger().log("entering the function");
        copierComponent.ldpDynamoToEsCopier().processMessages(sqsEvent);
        return null;
    }
}

Questions:

1. In case of success, LambdaHandler will just return null. Should it return anything else? Some kind of LambdaResponse with code 200? Some examples propose String "200 OK", but does it really matter?

2. In case of failure, processMessages will first remove processed messages from SQS, then raise an exception (BatchProcessingFailure). SQS will not receive any response, so after visibility timeout, it will return not processed messages back to the queue (there is a DLQ configured as well).

I don't like this method though. Is there some other method of returning some value or finishing lambda with a failed state? Some kind of LambdaResponse with code 500, that would inform SQS to not to remove messages and return them to queue?

Answer 1

1. No, you don't need to return anything special. The returned data is only relevant for other Lambda integrations, eg in combination with API Gateway.

2. I think there's no better way than throwing an exception in case of an error. From the AWS docs:

If your function successfully processes the batch, Lambda deletes the messages from the queue. If your function is throttled, returns an error, or doesn't respond, the message becomes visible again. All messages in a failed batch return to the queue, so your function code must be able to process the same message multiple times without side effects.

Source: Using AWS Lambda with Amazon SQS

My suggestions for you in case you don't want to reprocess messages multiple times:

1. Delete all successful messages (immediately) from the queue.
2. If one or more messages fail, throw an exception in your Lambda function. If you don't throw an exception, the Lambda service thinks everything went fine and deletes all messages of the batch from the queue - this is not what you want.

Answer 2

There seems to be support now for partial failures when consuming SQS messages from Lambda: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting