Rails: Uploading a file to AS3 using fog and resque

Question

The Problem:

I have a rails app that requires a user to upload some type of spreadsheet (csv, xslx, xsl etc) for processing which can be a costly operation so we've decided to send it off to a background service as a solution to this problem. The issue we're concerned about is that because our production system is on Heroku we need to store the file on AS3 first then retrieve later for processing.

Because uploading the file to AS3 is in itself a costly operation, this should probably also be done as a background job. The problem is the concern that using Resque to do this could eat up a lot of RAM due to Resque needing to put the file data into Redis or later retrieval. As you know, Redis only stores its data in RAM and also prefers simple key value pairs so we would like to try and avoid this.

Heres some pseudocode as an example of what we'd like try and do:

workers/AS3Uploader.rb

require 'fog'

class AS3Uploader
  @queue = :as3_uploader
  def self.perform(some, file, data)
    # create a connection
    connection = Fog::Storage.new({
      :provider                 => 'AWS',
      :aws_access_key_id        => APP_CONFIG['s3_key'],
      :aws_secret_access_key    => APP_CONFIG['s3_secret']
    })

    # First, a place to contain the glorious details
    directory = connection.directories.create(
      :key    => "catalog-#{Time.now.to_i}", # globally unique name
      :public => true
    )

    # list directories
    p connection.directories

    # upload that catalog
    file = directory.files.create(
      :key    => 'catalog.xml',
      :body   => File.open(blah), # not sure how to get file data here with out putting it into RAM first using Resque/Redis
      :public => true
  end

  # make a call to Enqueue the processing of the catalog
  Resque.enqueue(CatalogProcessor, some, parameters, here)
end

controllers/catalog_upload_controller.rb

def create
  # process params

  # call Enqueue to start the file processing
  # What do I do here? I could send all of the file data here right now
  # but like I said previously that means storing potentially 100s of MB into RAM
  Resque.enqueue(AS3Uploader, some, parameters, here)
end

Answer 1

The way I would suggest you to do would be

• store your file in tmp dir you create and get the file-path
• tell Resque to upload the file by using the file-path
• make Resque to store the file-path in the redis not the whole file-content ( It would be very expensive )
• Now worker will upload the file to AWS- S3

Note: If you have multiple instances like One instance for background processing, One for database, One as utility instance then your tmp dir may not be available to other instances.. so store the file in the temp dir inside the instance holding the resque