AWS - Sending 1000's of emails from Lambda / Node.js

Question

I have a "main" Lambda function that gets triggered by SNS. It pulls a list of recipients from the database and it needs to send each of them a message based on a template, replacing things like first name and such.

The way I have it setup is I created another Lambda function called "email-send" which is subscribed to "email-send" topic. The "main" Lambda then loops through the recipients list and publishes messages to "email-send" with a proper payload (from, to, subject, message). This might eventually need to process 1000's of emails in a single batch.

Is this a good approach to my requirements? Perhaps Lambda/SNS is not a way to go? If so, what would you recommend.

With this setup I am running into issues when my "main" function finishes running and somehow "sns.publish" does not get triggered in my loop. I assume because I am not letting it finish. But I am not sure how to fix it, being a loop.

Here is the snippet from my Lambda function:

exports.handler = (event, context, callback) => {
        // code is here to pull data into "data" array

        // process records
        for (var i = 0; i < data.length; i++) {
            var sns = new aws.SNS();
            sns.publish({
              Message: JSON.stringify({ from: data[i].from, to: data[i].to, subject: subject, body: body }),
              TopicArn: 'arn:aws:sns:us-west-2:XXXXXXXX:email-send'
            }, function(err, data) {
              if (err) {
                console.log(err.stack);
              } else {
                console.log('SNS pushed!');
              }
            });  
        }
        context.succeed("success");
};

Thanks for any assistance.

Answer 1

I think that a better approach is using AWS Lambda API.

That way, you don't need SNS.

For example:

var lambda = new AWS.Lambda({region: AWS_REGION});
function invokeWorkerLambda(task, callback) {
    var params = {
        FunctionName: WORKER_LAMBDA_NAME,
        InvocationType: 'Event',
        Payload: JSON.stringify({.....})
    };
    lambda.invoke(params, function(err, data) {
        if (err) {
            console.error(err, err.stack);
            callback(err);
        } else {
            callback(null, data);
        }
    });
}

As you can see, you don't need SNS for lambda function's invocation.

Important: Another suggestion is to create an Array of invocations ( functions ) and later execute them as follow:

async.parallel(invocations, function(err) {
    if (err) {
        console.error(err, err.stack);
        callback(err);
    }
});

Take a look at this link where I got a lot of knowledge about Lambda invocation: https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/

Answer 2

Your code is doing this...

1. Begin calling sns.publish() 1000 times
2. Return (through context.succeed() )

You didn't wait for those 1000 calls to finish!

What your code should do is...

1. Begin calling sns.publish() 1000 times
2. When all calls to sns.publish() has returned, then return . ( context.succeed is old so we should use callback() instead).

Something like this...

// Instantiate the client only once instead of data.length times
const sns = new aws.SNS();

exports.handler = (event, context, callback) => {
  const snsCalls = []
  for (var i = 0; i < data.length; i++) {
    snsCalls.push(sns.publish({
      Message: JSON.stringify({
        from: data[i].from,
        to: data[i].to,
        subject: subject,
        body: body
      }),
      TopicArn: 'arn:aws:sns:us-west-2:XXXXXXXX:email-send'
    }).promise();
  }

  return Promise.all(snsCalls)
    .then(() => callback(null, 'Success'))
    .catch(err => callback(err));
};