Sometimes data is not getting written in the AWS S3 bucket by Lambda

Question

I'm facing a freaking issue with my Lambda code written using TypeScript that creates S3 bucket and write an array of JSON data into it. This Lambda gets triggered based on messages arrived in the SQS queue. Sometimes, there can be many message all of a sudden.

I suspect that when there is just 1 message, then my Lambda works fine by first creating a S3 bucket and write array of JSON into it however when the messages grow say 10 messages at a time then Lambda just creates a bucket only and could not write the contents in it, as a result I just get an empty JSON in it like {}.

Not sure if it is due to no. of messages or not. Because all messages has to do same task that is creating same bucket (if not exists already) and write similar contents into or it has something to do CacheControl property describe below. Below is my code snippet :-

exports.createBucketAndUploadToS3 = async (s3Client, bucket, prefix, contents) => {
const params = {
Bucket: bucket,
Key: `${prefix}/data.json`,
Body: JSON.stringify(contents),
ContentType: 'application/json; charset=utf-8',
CacheControl: 'max-age=60'
};
await s3Client.createBucket({ Bucket: bucket }).promise();
return await s3Client.putObject(params).promise();
};

Answer 1

I would propose to check if the bucket exists before calling the createBucket command, this often causes an exception when you try to create an existing object again. You can do this using the following code:

const checkBucketExists = async bucket => { 
  const s3 = new AWS.S3();
  const options = {
    Bucket: bucket,
  };
  try {
    await s3.headBucket(options).promise();
    return true;
  } catch (error) {
    if (error.statusCode === 404) {
      return false;
    }
    throw error;
  }
};

// in your code

let isBucketExisting = await checkBucketExists(bucket);

if (isBucketExisting) {
 await s3Client.createBucket({ Bucket: bucket }).promise();
}

return await s3Client.putObject(params).promise();