Errno::ETIMEDOUT: Connection timed out - connect(2)

Question

In production, I'm occasionally getting the following error:

Errno::ETIMEDOUT: Connection timed out - connect(2)

It only seems to be happening when I generate a PDF using the prawn gem that contains an image that was uploaded by paperclip/aws-sdk to s3. This probably only happens a couple times a week on an action that is used hundreds of times a day (with no issues), but it results in a 500 error when it does fail.

The trace is:

/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`initialize'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`open'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`connect'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/timeout.rb:53:in
`timeout'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/timeout.rb:101:in
`timeout'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`connect'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:553:in
`do_start'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:542:in
`start'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:242:in
`open_http'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:616:in
`buffer_open'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:164:in
`open_loop'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:162:in
`catch'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:162:in
`open_loop'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:132:in
`open_uri'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:518:in
`open'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:30:in
`open'
....rb:57:in `render_image_to_pdf'

Line 57:

pdf.image open(image.expiring_url(30.minutes, :full)), :width => 300, :position => 20

Setup:

Rails 3.0.10
Ruby 1.8.7EE
Prawn  0.11.1
AWS-SDK 1.3.3
Paperclip 2.5.2

What can I do to prevent this error?

Answer 1

One option is to catch the exception and try again:

begin
  pdf.image open(image.expiring_url(30.minutes, :full)), :width => 300, :position => 20
rescue Errno::ETIMEDOUT
  # try one more time, or use retry with a counter to attempt a limited number of times
  pdf.image open(image.expiring_url(30.minutes, :full)), :width => 300, :position => 20
end

You may also want to put some diagnostics (either logging or generating an Airbrake) that notifies you so you can see if there is some sort of pattern to the failures.

Answer 2

I have been getting the same error in Amazon EC2 when using aws-sdk to spawn new server instances. Here's what I know:

1. I have a function (sleep_while_not_running) which tests the instance.status:

   def sleep_while_not_running(instance) print "waiting on system: "
   $stdout.flush while instance.status != :running print "." $stdout.flush sleep 1 end puts "resuming." end

2. After sleep_while_not_running returns, I assume that we should be able to connect (as follows):

    Net::SSH.start(instance.ip_address,'root',:key_data=>[key_pair.private_key]) do |ssh| sleep_while_not_running(instance) puts "testing SSH connectivity." puts ssh.exec("uname -a") puts " " 

3. What happens is the same time-out error you describe. Investigating manually, I have found that--

   a. If I wait until "Status Checks" (in the management console) shows that the new instance has "Passed 2/2 Checks", then I can SSH in manually just fine using the same key pair used by the script.

   b. sleep_while_not_running checks instance.status (which says "running" long before the server is fully baked.

   c. I can find no call in aws-sdk to check the "status checks" I would check in the management console, but I suspect that is where the solution exists.

PS I realize this isn't an "answer" but it may help identify a solution.