How do download S3 file to Lambda disk, append to it, and upload back to S3

Question

The goal is to be able to do this without loading the potentially large file into memory on a Lambda with limited memory available. I just want to make sure I'm doing this right:

const stream = fs.createWriteStream(`/tmp/${key}`, {flags:'a'});
const download = async (args, stream) => {
  const params = {
    Key: 'fileKey',
    Bucket: 'filebucket'
  };
  const readStream = S3.getObject(params).createReadStream();
  await readStream.pipe(stream);
};

Answer 1

The pipe() method directly passes the read stream to the write stream, so to my understanding the object doesn't need to be stored in the Lambda function's memory. Your piece of code would likely work as intended.

However, the size of the /tmp directory, which is dedicated for data storage on AWS Lambda, has a hard limit of 512 MB . If this is sufficient for your use case, you can store the object. If your file size could possibly exceed 512 MB, you will have to use a different storage mechanism, like eg Amazon EFS .

Note: The Lambda function's memory configuration also has an effect on the bandwidth that's available to your function. So increasing the memory could actually make the download/upload of your S3 object much faster and make the whole process more cost-effective than using the bare minimum amount of memory.

This is something you definitely should experiment with. To try different configurations and find the sweet spot of speed and cost-effectiveness, you can use the AWS Lambda Power Tuning tool .