Is it possible to pipe a file upload using ruby?
Instead of reading the whole file into memory, or writing a temp file to then write to S3, I was hoping to stream the request body directly.
Streaming data to S3 with ruby shows how to stream to S3, but I'm not sure how to turn the request body in a rack app into a readable stream.
class FastUploadApp
def self.call(env)
# what goes here?
end
end
You're exactly right that you need to turn the request body into a readable stream. Specifically, S3 expects a Ruby IO class (in that it wants a #read method and a #eof? method). Rack request bodies don't have #eof? defined, however, so you have to make a little wrapper class:
class RackS3Wrapper
def initialize(body)
@body = body
@eof = false
end
def read(*args)
ret = @body.read(*args)
if ret == nil or ret == ""
@eof = true
end
ret
end
def eof?
@eof
end
end
Then you can use this wrapper to stream the request to S3 directly:
s3.buckets['com.mydomain.mybucket'].objects['filename'].write(
:data => RackS3Wrapper.new(request.body),
:content_length => request.env['CONTENT_LENGTH'].to_i)
This hasn't been tested in production or anything, but it should work fine.
Check out my project middl: https://github.com/diclophis/middl
It has some caveats ... mostly you have to rely on the client sending an appropriate request size header (which is not required by the w3 spec)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.