使用 API 网关将大型文件上传/下载到 Lambda function 而不使用任何 S3 存储桶

Question

I'm implementing a serverless API , using:

1. API Gateway
2. Lambda
3. S3 Bucket "If needed"

My flow is to:

1. Call POST or PUT method with a binary file "zip", upload it to Lambda.
2. In Lambda: unzip the file.
3. In Lambda: Run a determined script over the extracted files.
4. In Lambda: Generate a new zip.
5. Return it to my desktop.

This flow is already implemented and it's working well with small files, 10MB for uploading and 6MB for downloading. But I'm getting issues when dealing with large files as it'll be the case on many occasions. To solve such issue I'm thinking about the following flow:

1. Target file gets uploaded S3 Bucket.
2. A new event is generated and Lambda gets triggered.

3. Lambda Internal Tasks:

   3.1 Lambda Download the file from S3 bucket.

   3.2 Lambda Generate the corresponding WPK Package.

   3.3 Lambda Upload the generated WPK package into S3.

   3.4 Lambda returns a signed URL related to the uploaded file as a response.

But my problem with such design is that it requires more than a request to get completed. I want to do all this process in only 1 request, passing the target zip file in it and get the new zip as the response. Any Ideas, please?

My Components and Flow Diagram would be:

Component and Flow Diagram

Answer 1

There are a couple of things you could do if you'd like to unzip large files while keeping a serverless approach:

1. Use Node.js for streaming of the zip file, unzipping the file in a pipe, putting the content in a write stream pipe back to S3.
2. Deploy your code to an AWS Glue Job.
3. Upload the file to S3,AWS Lambda gets triggered pass the file name as the key to the glue job and the rest will be done. This way you have a serverless approach and a code that does not cause memory issues while unzipping large files