S3 alternative that allows upload-stream files without buffering

Question

Is there a better online storage service than Amazon's S3, which requires multipart uploads and the whole file to be buffered to my server before it's uploaded to them.

I want some service that I can directly stream uploads to it (through my server) without any buffering

Answer 1

Assuming that what you have is a complete file that you want to end up stored on Amazon, there isn't much else you can do.

You can do streaming to S3 by using the low-level API: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectOps.html

The only alternatives are to transfer things piece-wise and then later reassemble them. For example, you could use Kinesis Firehose to upload individual file-chunks to S3. Then you'd need some other job to come along and mash the pieces back together into the original file.

Answer 2

You don't have to buffer the entire file to your server before uploading to S3. S3's multipart upload allows you to upload each part separately, which can be as small as 5MB, ie: you server only has to buffer 5MB at a time. I use this technique in goofys to simulate streaming writes.