Amazon s3: direct upload vs presigned url

Question

I'm using AWS SDK for .NET and I was looking for a method to let user upload directly to a s3 storage.

I've come across two different ways offedered by aws:

Browser based upload: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html

and presigned urls: https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjectPreSignedURLDotNetSDK.html

It seems presigned url is 'easier' since a method is already present within aws sdk to generate a url to pass to client, to let him PUT object directly to the bucket (and it seems also painless compared to browser upload, since it doesn't require all the keys browser upload wants in the post form).

But I was wondering why there are actually two different methods. What are PRO and CONS of each one?

Answer 1

It totally comes down to the point, whether you want to use the REST API or the AWS SDKs to interact with S3.

In both cases, you need to prove(authenticate/Sign-Request) your identity unless bucket is public.

a) If you are going with REST APIs, to prove identity, you need sign your request using ' AWS Signature version 4 ' (deprecated ver 2 is also there), which includes three methods (one you have listed)

1. Authenticating Requests: Using the Authorization Header (AWS Signature Version 4)
2. Authenticating Requests: Using Query Parameters (AWS Signature Version 4)
3. Authenticating Requests: Browser-Based Uploads Using POST (AWS Signature Version 4)

b) If you are going to use AWS SDKs, you should let SDK do the signing ceremony(process). So the choice is straightforward to use SDK to sign the request

(Part of the question) It seems also painless compared to browser upload since it doesn't require all the keys browser upload wants in the post form <

For below code, s3Client already has got your creds whether from AWS-CLI-Profile(if running local/laptop), IAM Role(in case of EC2, lambda, etc)

string url = s3Client.GetPreSignedURL(request);

Answer 2

The main difference that I have found is memory consumption. If you upload the same image (46KB) 200 times (as a test/benchmark) the memory consumption differs wildly.

PutObject with any version of AWSSDK.Core below 3.3.21.19 takes 116MB (SOH: 66MB / LOH: 50MB )

PutObject with any version of AWSSDK.Core above 3.3.21.19 takes 99MB (SOH: 98MB / LOH: 0.4MB )

You can further reduce the memory consumption if you set UseChunkEncoding to false on PutObjectRequest :-

PutObject with UseChunkEncoding set to false takes 52MB (SOH: 52MB / LOH: 0.4MB )

But GetPreSignedURL is still better from a memory point of view;

GetPreSignedURL takes 32MB (SOH: 32MB / LOH: 0.4MB )

If you are calling PutObject a lot then it may be benefical to switch out to GetPreSignedURL to save memory if that's a problem. The downside is that you are responsible for retrying mechanisms etc.

We use PutObject everywhere in our code base except one place where the code path is very hot and is hit around 800,000+ times a day. In that case it made sense to switch it out to GetPreSignedURL .

Hope that helps!