AWS S3 Muitipart Upload via API Gateway or Lambda

Question

I'm trying to create a reusable large-file serverless upload service in AWS (we host a number of sites). What I would like to do is to set up an API Gateway in AWS and use CORS to control which sites can upload, allowing the sites to use client-side code. Here is what I've tried and the roadblocks I've run into. Wondering if anybody has any suggested workarounds?

• Calling S3 from client-code upload would require me to expose authentication information in client-side land, which seems bad
• API Gateway does not appear to support calling S3 multipoint through its AWS Service integration type (URL is fixed to generic S3 service URL, and IAM isn't supported in HTTP integration type)
• Leveraging Lambda to call the multipart API won't work, because it can only take in 6 MB of invoke request payload, and to get the 5 MB worth of minimal upload part size, base64 will make the data way more than 6 MB
• I could do my own partial upload functionality in Lambda, storing the chunks in S3, but I can't figure out how to merge them together within Lambda's memory and tmp storage space (still PassThrough streams do not appear to work with AWS SDK)

Any ideas? Is any of these worth digging into? Or is serverless a no-go for this use case?

So, after further follow-up with Amazon, it's sort-of possible to use pre-signed URLs with the multipart API, but it's not very practical. Steps involved would include the following:

1. Create a new file, and split it into parts.
2. Generate a presigned URL to initiate the multiart upload.
3. Use the presigned URL to initiate the upload.
4. Generate a presigned URL for each part, using a part number.
5. Use the URLs to send the PutPart requests. Keep track of the Etag that is returned for the part number.
6. Combine all of the parts and corresponding ETAGs to form the request body.
7. Generate a presigned URL to complete the MP upload.
8. Complete the multipart upload by sending the request with the presigned complete multipart upload URL.

Will accept Angelo's answer since it did point in this direction which, technically, seems possible

Answer 1

You might be able to use presigned urls for the upload. In this case the client would hit your API, which would do whatever validation is necessary, and then generated a presigned url to S3 that is returned to the client. The client then directly uploads to s3.

You can see some information here: https://sanderknape.com/2017/08/using-pre-signed-urls-upload-file-private-s3-bucket/