S3直接上传性能

Question

So I'm faced with a choice of whether to implement a direct upload to S3 or to proxy images through my servers and then to S3.

I do need to process images once they are uploaded, creating thumbs and different versions.

So, in short what I would like to know is which approach is better, from those that might have done this before.

Choice 1. Direct upload to S3

• User uploads to S3
• Background worker downloads the new image from S3
• Image is processed
• Images are uploaded to S3

Choice 2. Upload through servers

• User uploads to server
• Image is processed
• Images are uploaded to S3

Looking at this, it would seem that uploading directly to S3 is actually slower than through my servers first. However, does the load that initial upload places on server outweigh the extra download step when going directly to S3?

Also are there additional factors involved, such as network performance in each case and similar?

Answer 1

Regards to performance, uploading directly to S3 is a better solution. The reason is that S3 is highly scalable, which means: if you use one thread to upload files to S3, the speed is 1M/s, if you use 10 threads, the speed will be about 10M/s.

For you problem, if you have a lot of users uploading files at the same time, S3 can handle these requests with less performance personality.As uploading files directly to the server, it requires your server can handle many requests at the same time, if your implementation is salable, it may have a chance to beat S3 on performance.

But solution 1 has more costs as it has more transactions, and also data transfer out.

Answer 2

One issue with uploading directly to S3 is that you're uploading to the location of the bucket, whereas your servers may be distributed across multiple locations throughout the world. (For example, if you're running your servers in multiple EC 2 regions). In that case, you find you have lower latency (higher transfer rates) going to your server than going to S3 directly.

Also, there may not be transfer out charges from S3 if you're transferring data to a server running on EC2 in the same region. Otherwise Matt's right - you'll pay anywhere from $0.02 - $0.25 / GB for transferring data from S3.

Mike

Answer 3

I was a member of a big project using S3 storage.

S3 upload is not 100% perfect officially.

It is very important that whether you choose one of them you mentioned. Because each one has different benefits and drawbacks.

You should choose first one if you want to make overload in client side instead of server side because of some reason like 1. Server is low spec or busy 2. There are huge upload files in huge clients side. 3. Customer want to know the result of upload immediately and want to do next action with uploaded files. But you should make strong error handling logins in upload program. Otherwise, you might be suffered from customer's complain.

.

You should choose second one if you want to have higher reliability of uploading process in terms of structure or development environment. Because S3 uploading is HTTP protocol so it is very slow and sometime it may make overload in upload side. If there are some abnormal cases in client side like stopping upload or network problem, you might get into problem handling the situation. So it is stable to upload files in server side again because error handling is essential as amazon mentioned below link.

http://docs.aws.amazon.com/AmazonS3/latest/dev/ErrorBestPractices.html The URL mentioned "When designing an application for use with Amazon S3, it is important to handle Amazon S3 errors appropriately ..."

Also I recommend you use just only official APIs from Amazon to handle errors instead of plug-in or S3 application. :)