Performance issues uploading large text files via Paperclip

Question

I'm in the process of upgrading from Ruby 1.8.7 to 1.9.3 and from Rails 2.3 to 3.2 As part of that upgrade, I'm moving from Paperclip 2.2.9 to 3.5.2. My ImageMagick version is 6.8.6. One issue that I've discovered as part of the upgrade process is that upload performance is very poor when it comes to large (~1 MB) text files. The files in question do not need to specifically be .txt files, anything in plain text format (.xml files, for example) are also effected.

For your reference, here is my Paperclip setup:

  has_attached_file :attachment,
    :url => "/shared_documents/:id/:basename.:extension",
    :path => ":rails_root/user_uploaded_content/shared_documents/:id/:basename.:extension"

For simplicity, I'm omitting our validations and so on as we are simply checking file size and presence.

Watching the top processes running on my development machine, it seems that the bottleneck occurs when Paperclip is calling ImageMagick's identify command. Calling identify on a variety of files through the command line has allowed me to verify that metadata is returned almost immediately for image files but large, non-image text files take a very long time to process.

For my application, I am allowing users to upload documents in whatever format they like so I must be able to efficiently process both images and text files. Has anyone else encountered this issue? Is there a way to selectively disable calling identify on certain file formats in Paperclip but not others? Failing that, we could live with simply not calling identify if that is an option. Perhaps there a way to set configure ImageMagick to more gracefully handle large text files?

Answer 1

If you're not actually post processing the files, just tell Paperclip not to post process them. From the Paperclip documentation, you can do this a couple of ways. One is to supply an empty list of styles in the model:

has_attached_file :attachment, 
  styles:{}, 
  url: "/shared_documents/:id/:basename.:extension",
  path: ":rails_root/user_uploaded_content/shared_documents/:id/:basename.:extension"

or, you may just supply no processors

has_attached_file :attachment, 
  processors:[], 
  url: "/shared_documents/:id/:basename.:extension",
  path: ":rails_root/user_uploaded_content/shared_documents/:id/:basename.:extension"

or, you could possibly use the before_post_process callback in your model and return false to halt the process, but Paperclip may call identify first to validate the file, which would make this option pointless for your situation:

has_attached_file :attachment, 
  url: "/shared_documents/:id/:basename.:extension",
  path: ":rails_root/user_uploaded_content/shared_documents/:id/:basename.:extension"

before_post_process :skip_processing

def skip_processing
  false
end