Capistrano : NoMethodError: undefined method `recv' for #<IO:fd XX>

Question

This error in Capistrano (> 3) prevents me from deploying my Rails application. My Capistrano config works fine on another environment.

DEBUG [6c7d7bed] Command: if test ! -d /opt/my_app/releases/20140429175529; then echo "Directory does not exist '/opt/my_app/releases/20140429175529'" 1>&2; false; fi
cap aborted!
NoMethodError: undefined method `recv' for #<IO:fd 49>
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/buffered_io.rb:65:in `fill'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/packet_stream.rb:86:in `next_packet'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:178:in `block in poll_message'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:173:in `loop'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:173:in `poll_message'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:210:in `block in wait'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:208:in `loop'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:208:in `wait'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:87:in `initialize'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh.rb:200:in `new'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/net-ssh-2.8.0/lib/net/ssh.rb:200:in `start'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/connection_pool.rb:24:in `create_or_reuse_connection'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:173:in `ssh'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:126:in `block in _execute'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:123:in `tap'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:123:in `_execute'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:66:in `execute'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/abstract.rb:75:in `within'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/capistrano-3.2.1/lib/capistrano/tasks/deploy.rake:207:in `block (3 levels) in <top (required)>'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:54:in `instance_exec'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:54:in `run'
/Users/such/.rvm/gems/ruby-2.1.1@my_app/gems/sshkit-1.4.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => deploy:set_current_revision
(See full trace by running task with --trace)
The deploy has failed with an error: #<NoMethodError: undefined method `recv' for #<IO:fd 49>>

I'm using a proxy to connect to my remote servers (on both environment). SSH connection is working perfectly.

set :ssh_options, proxy: Net::SSH::Proxy::Command.new('ssh user@gateway -W %h:%p')

Has anyone ever experienced this issue?

Workaround

My weak understanding of the error is that after the ssh negotiation, sometimes the socket returned to net-ssh is a file descriptor which is unexpected. I have added:

gem 'sshkit', github: 'msimonin/sshkit', branch: 'gateway'

to my Gemfile so that the ssh connection is only established once.

Related configuration is:

SSHKit.config.backend.configure do |ssh|
  ssh.gateway = 'user@gateway'
end

It works fine this way. I can't wait for the pull request to be merged.

Answer 1

The connection objects are always a file descriptor-backed IO instance if you're using proxy commands. The code lives in lib/net/ssh/proxy/command.rb and it injects the recv method right after the bit that popens the pipes and waits for something to become readable, so I'm not sure why it sometimes fails (I've seen it too). Based on the callstack, I think it has returned already, so recv should be there.

Have you opened an issue?