Stream remote file to client in ruby/rails 4/unicorn/nginx

Question

I am trying to stream a file from a remote storage service (not s3 :-)) to the client using Ruby on Rails 4.2.

My server needs to stay in the middle of things to authenticate the client request but also to build up the request to the remote storage service since all requests to that service need to be authenticated using a custom header param. This makes it not possible to do a simple redirect_to and let the client download the file directly (but do let me know if this IS in fact possible using rails!). Also I want to keep the url of the file cloaked for the client.

Up until now I am using a gem called ZipLine but this also does not work as it still buffers the remote file before sending it to the client. As I am using unicorn/nginx, this might also be due to a setting in either of those two, that prevents proper streaming.

As per rails doc's instructions I have tried adding

listen 3000, tcp_nopush: false

to config/unicorn.rb but to no avail.

A solution might be to cache the remote file locally for a certain period and just serve that file. This would make some things easier but also creating new headaches like keeping the remote and cached files in sync, setting the right triggers for cache expiration, etc.

So to sum up:

1) How do I accomplish the scenario above?

2) If this is not a intelligent/efficient way of doing things, should I just cache a remote copy?

3) What are your experiences/recommendations in given scenario?

I have come across various solutions scattered around the interweb but none inspire a complete solution.

Thanks!

Answer 1

I am assuming you the third party storage service has an HTTP access. If you did consider using redirect_to , I assume the service also provides a means to allow per download authorization. Like unique key in the header that expires and does not expose your secret api keys or HMAC signed URL with expiration time as a param.

Anyhow, most cloud storage services provide this kind of file access. I would highly recommend let the service stream the file. Your app should simply authorize the user and redirect to the service. Rails allows you to add custom headers while redirecting. It is discussed in Rails guides .

10.2.1 Setting Custom Headers

If you want to set custom headers for a response then response.headers is the place to do it. The headers attribute is a hash which maps header names to their values, and Rails will set some of them automatically. If you want to add or change a header, just assign it to response.headers

So your action code would end up being something like this:

def download
  # do_auth_check

  response.headers["Your-API-Auth-Key"] = "SOME-RANDOM-STRING"
  redirect_to url
end

Don't use up unnecessary server resources by streaming through them all those downloads. We are paying cloud services to that after all :)