Need help configuring DLQ for Lambda triggered by SNS

Question

I'd like to receive an email if my Lambda fails. The Lambda is triggered via SNS (which is triggered by SES).

When I publish to the SNS Topic, the Lambda runs and throws an error (for testing) due to a missing package. I see from the console logs that the Lambda runs 3 times.

I have an SQS queue attached to the Redrive policy (dead-letter queue) of the SNS Topic's subscription (that triggers the lambda).

{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:myproj-sns-topic-dlq"
}

I tested, and things didn't work. I noticed a warning in the AWS console for the SNS Topic's subscription:

Dead-letter queue (redrive policy) permissions The Amazon SQS queue specified as the dead-letter queue for your subscription (redrive policy) doesn't permit deliveries from topics. To allow an Amazon SNS topic to send messages to an Amazon SQS queue, you must create an Amazon SQS queue policy.

Following the steps Subscribing an Amazon SQS queue to an Amazon SNS topic , I added the 2nd statement to my SQS queue's Access policy :

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:myproj-sns-topic-dlq"
    },
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:myproj-sns-topic-dlq",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:myproj-snstopic"
        }
      }
    }
  ]
}

The Principal was {"Service": "sns.amazonaws.com"} , but that results in a warning in the AWS console saying it can't test permissions. I tested anyway and it didn't work. (Lambda runs 3 times, but nothing gets put in the DLQ.)

I set the Principal to * for now (per snippet above). That eliminates the warning in the console, but things still don't work.

My goal it to have the event drop into the SQS DLQ after the Lambda fails. I have an alarm on that queue that will notify me by email...

Edit: added missing condition

Answer 1

According to this article , you can use a CloudWatch Log filter to parse a log for a Lambda function and get an email notification.

To implement this solution, you must create the following:

• An SNS topic
• An IAM role
• A Lambda function
• A CloudWatch log trigger

Answer 2

As pointed out by @fedonev, the SNS Subscription's (for lambda) DLQ is used when the event cannot be delivered. If the event is delivered (but the Lambda fails), you can use Lambda's async event DLQ or wire-up the 'on failed' destination of the Lambda.

I'm using AWS Amplify and decided to use the Lambda's "async" DLQ as opposed to a lambda destination.

Step 1 - Add a custom category to add:

• SQS (dlq) to save the failed attampt's event
• CloudWatch Alarm to watch the SQS resource
• SNS Topic and Subscription(s) used by the Alarm

And "Output" the SQS queue's ARN which is needed by the Lambda.

Step 2 - Add a "DeadLetterConfig" to the Lambda that pushes fails into the above queue.

amplify add custom

name: LambdaAlarm

File: amplify/backend/custom/LambdaAlarm/LambdaAlarm-cloudformation-template.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Parameters": {
    "env": {
      "Type": "String"
    }
  },
  "Resources": {
    "SQSDLQ": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "QueueName": {
          "Fn::Join": [
            "",
            [
              "myproject-lambdafailed-dlq",
              {
                "Fn::Select": [
                  3,
                  {
                    "Fn::Split": [
                      "-",
                      {
                        "Ref": "AWS::StackName"
                      }
                    ]
                  }
                ]
              },
              "-",
              {
                "Ref": "env"
              }
            ]
          ]
        },
        "MessageRetentionPeriod": 1209600,
        "VisibilityTimeout": 5432,
        "SqsManagedSseEnabled": false
      }
    },
    "SNSTOPIC": {
      "Type": "AWS::SNS::Topic",
      "Properties": {
        "TopicName": {
          "Fn::Join": [
            "",
            [
              "myproject-lambda-failed-alarm-topic",
              {
                "Fn::Select": [
                  3,
                  {
                    "Fn::Split": [
                      "-",
                      {
                        "Ref": "AWS::StackName"
                      }
                    ]
                  }
                ]
              },
              "-",
              {
                "Ref": "env"
              }
            ]
          ]
        }
      }
    },
    "SNSSubscriptionEmailJoeAtGmail": {
      "Type": "AWS::SNS::Subscription",
      "Properties": {
        "Protocol": "email",
        "TopicArn": {
          "Ref": "SNSTOPIC"
        },
        "Endpoint": "yourname+myprojectalert@gmail.com"
      }
    },
    "SNSSubscriptionEmailJillAtQuad": {
      "Type": "AWS::SNS::Subscription",
      "Properties": {
        "Protocol": "email",
        "TopicArn": {
          "Ref": "SNSTOPIC"
        },
        "Endpoint": "jill@stakeholder.com"
      }
    },
    "ALARM": {
      "Type": "AWS::CloudWatch::Alarm",
      "Properties": {
        "AlarmName": {
          "Fn::Join": [
            "",
            [
              "myproject-lambda-failed-dlq-alarm",
              {
                "Fn::Select": [
                  3,
                  {
                    "Fn::Split": [
                      "-",
                      {
                        "Ref": "AWS::StackName"
                      }
                    ]
                  }
                ]
              },
              "-",
              {
                "Ref": "env"
              }
            ]
          ]
        },
        "AlarmDescription": "There are messages in the 'Lambda Failed' dead letter queue.",
        "Namespace": "AWS/SQS",
        "MetricName": "ApproximateNumberOfMessagesVisible",
        "Dimensions": [
          {
            "Name": "QueueName",
            "Value": {
              "Fn::GetAtt": [
                "SQSDLQ",
                "QueueName"
              ]
            }
          }
        ],
        "Statistic": "Sum",
        "Period": 60,
        "EvaluationPeriods": 1,
        "Threshold": 0,
        "ComparisonOperator": "GreaterThanThreshold",
        "AlarmActions": [
          {
            "Ref": "SNSTOPIC"
          }
        ]
      }
    }
  },
  "Outputs": {
    "SQSDLQArn": {
      "Value": {
        "Fn::GetAtt": [
          "SQSDLQ",
          "Arn"
        ]
      }
    }
  },
  "Description": ""
}

Next, update and add the new custom resource as a dependency of the Lambda(s) to monitor.

File: backend-config.json

"function": {
  "MYLambda": {
    "build": true,
    "dependsOn": [
      {
        "attributes": [
          "SQSDLQArn"
        ],
        "category": "custom",
        "resourceName": "LambdaAlarm"
      }
    ],
    "providerPlugin": "awscloudformation",
    "service": "Lambda"
  },
},

In the Lambda(s) you want to monitor, make 3 changes to the cloudformation:

1. Pull in the output variable ( customLambdaAlarmSQSDLQArn ) from your custom category and add it to the Parameters
2. Add the DeadLetterConfig property to the Lambda
3. Add a policy to the LambdaExecutionRole

File: amplify/backend/function/MyLambda/MyLambda-cloudformation-template.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "...",
  "Parameters": {
    ...snip...
    "customLambdaAlarmSQSDLQArn": {
      "Type": "String"
    },
    ...snip...
  },
  "Conditions": {...snip...},
  "Resources": {
    "LambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Metadata": {...snip},
      "Properties": {
        "Code": {...snip...},
        "Handler": "index.handler",
        "FunctionName": {...snip...},
        "Environment": {
          "Variables": {...snip...}
        },
        "Role": {...snip...},
        "Runtime": "nodejs18.x",
        "Architectures": ["arm64"],
        "Layers": [],
        "MemorySize": 256,
        "Timeout": 120,
        "DeadLetterConfig": {
          "TargetArn": {
            "Ref": "customLambdaAlarmSQSDLQArn"
          }
        }
      }
    },
    "LambdaExecutionRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": {...snip...},
        "Policies": [
          {
            "PolicyName": "custom-lambda-execution-policy",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Sid": "AllowSQSSendMessage",
                  "Effect": "Allow",
                  "Action": [
                    "SQS:SendMessage"
                  ],
                  "Resource": {
                    "Ref": "customLambdaAlarmSQSDLQArn"
                  }
                }
              ]
            }
          }
        ],
        "AssumeRolePolicyDocument": {...snip...}
      }
    },
    "lambdaexecutionpolicy": {...snip...},
    "AmplifyResourcesPolicy": {...snip...},
    "CustomLambdaExecutionPolicy": {...snip...}
  },
  "Outputs": {...snip...}
}

Finally, due to an Amplify quirk you must amplify env checkout dev because you manually touched the backend-config.json file.

Then you can deploy your changes. The above is not specific to AWS Amplify.