How to send data to a client from a Resque worker?

Question

I've got a rails app running on Heroku. I'm trying to add a feature that lets users download some of their data in CSV format. The CSV files are largeish (~ 1-2 MB) and they're slow to generate. So as not to tie up my web dynos, I'm passing these download request off to worker dynos (using Resque), and assembling the CSV there. My resque worker looks something like this.

class Downloader

  def self.perform(ids_to_download)
      models = Model.where(:id => ids_to_download)
      csv_data_for_client = models.to_csv 
  end

end

I'm wondering about the best way to send the csv file back to the client who requested it.

If this code were in a Rails controller, this could be accomplished with simply send_data csv_data_for_client . This approach doesn't seem to work from within the Resque worker, though.

From researching on SO and elsewhere on the internet, it seems that I have a few options.

1. Send the data directly from the Resque worker ( Though I'm not sure if this is possible)
2. Save the data to the heroku filesystem, retrieve it in a controller, and then send it with send_data
3. Save it to the database (postgresql), retrieve it in a controller, and then use send_data

Can anyone offer advice on the best way to proceed? Thanks.

Answer 1

After doing some thinking, I eliminated options 1 and 2.

Re 1, I couldn't figure out how to send the csv file directly to the client from the Resque worker.

Re 2, saving the csv file to Heroku didn't seem possible either. Heroku's docs say that each dyno has its own filesystem , and this file system is only visible to the dyno that owns it. My worker dyno can't read or write to my web dyno's filesystem, and vice versa. So this didn't seem like a promising option either.

That left me with option 3, saving the csv file to my database in my worker dyno, and then retrieving it and sending it to the client in my web dyno.

I used the resque-status gem to assign a unique id (uuid) to each csv file request.

The I created a simple Model to hold the generated csv file, and associate it with its uuid.

class DownloaderJobOutput < ActiveRecord::Base
  attr_accessible :file_to_download, :uuid
end

In my resque worker, after building the csv file, I saved it to my database.

class Downloader
  @queue = :trial_downloader_queue
  include Resque::Plugins::Status

  def perform
    models = Model.where(:id => options["trial_ids"])
    csv = models.to_csv
    DownloaderJobOutput.create!( :uuid => self.uuid, :file_to_download => csv )
  end
end

While the job was executing, I polled it from my web client to keep track of its status. When it completed, I pulled it out the database and send it to the client.