无法在capistrano中部署多个主机

Question

I'm using capistrano to deploy rails app to about 20 servers, it works ok when server num ~ 10, when increase it to >~ 10, it doesnt work anymore

Here is the error

cap aborted!
Errno::EPIPE: Broken pipe
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/server_version.rb:44:in `write'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/server_version.rb:44:in `negotiate!'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/server_version.rb:32:in `initialize'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:84:in `new'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:84:in `initialize'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh.rb:232:in `new'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh.rb:232:in `start'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/sshkit-1.11.2/lib/sshkit/backends/connection_pool.rb:59:in `call'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/sshkit-1.11.2/lib/sshkit/backends/connection_pool.rb:59:in `with'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/sshkit-1.11.2/lib/sshkit/backends/netssh.rb:155:in `with_ssh'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/sshkit-1.11.2/lib/sshkit/backends/netssh.rb:49:in `upload!'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/capistrano-3.6.1/lib/capistrano/tasks/git.rake:24:in `block (3 levels) in <top (required)>'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/sshkit-1.11.2/lib/sshkit/backends/abstract.rb:29:in `instance_exec'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/sshkit-1.11.2/lib/sshkit/backends/abstract.rb:29:in `run'
/Users/khiem-nguyen/workspace/ruby/nenga-onepiece/cl-chef/deploy/vendor/bundle/ruby/2.3.0/gems/sshkit-1.11.2/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => git:check => git:wrapper
The deploy has failed with an error: Broken pipe
** Invoke deploy:failed (first_time)
** Execute deploy:failed    


** DEPLOY FAILED
** Refer to log/capistrano.log for details. Here are the last 20 lines:    


 DEBUG Uploading /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh 0.0%    

 DEBUG Uploading /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh 0.0%    

  INFO [7c256e12] Running /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh on onepiece-web07    

 DEBUG [7c256e12] Command: ( export RBENV_ROOT="$HOME/.rbenv/bin/rbenv" PATH="$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH" ; /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh )    

  INFO Uploading /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh 100.0%    

  INFO Uploading /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh 100.0%    

  INFO [846397ba] Finished in 0.065 seconds with exit status 0 (successful).    

  INFO [3e8f8a49] Running /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh on onepiece-web03    

 DEBUG [3e8f8a49] Command: ( export RBENV_ROOT="$HOME/.rbenv/bin/rbenv" PATH="$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH" ; /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh )    

  INFO [7c256e12] Finished in 0.037 seconds with exit status 0 (successful).    

  INFO [264aa0db] Running /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh on onepiece-web01    

 DEBUG [264aa0db] Command: ( export RBENV_ROOT="$HOME/.rbenv/bin/rbenv" PATH="$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH" ; /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh )    

  INFO [a8720934] Finished in 0.265 seconds with exit status 0 (successful).    

  INFO [3e8f8a49] Finished in 0.043 seconds with exit status 0 (successful).    

  INFO [264aa0db] Finished in 0.038 seconds with exit status 0 (successful).    

 DEBUG Uploading /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh 0.0%    

  INFO Uploading /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh 100.0%    

  INFO [3d959fa8] Running /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh on onepiece-web02    

 DEBUG [3d959fa8] Command: ( export RBENV_ROOT="$HOME/.rbenv/bin/rbenv" PATH="$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH" ; /usr/bin/env chmod 700 /tmp/git-ssh-nenga-onepiece-production-khiem-nguyen.sh )    

  INFO [3d959fa8] Finished in 0.045 seconds with exit status 0 (successful).

I think there's something related to ssh max connection???

Cap version:3.6.1 SSH kit: 1.9.0

Answer 1

SSHKit (the SSH layer in Capistrano) uses connection pooling. This means that it maintains an SSH connection for each host. If you are seeing a broken pipe error when you increase the hosts, the only thing I can think of is that perhaps a firewall or gateway (ie an ssh proxy) is limiting the number of simultaneous connections.

As a workaround, you can try disabling the connection pool. Add the following to your deploy.rb :

# Disable connection pool
SSHKit::Backend::Netssh.pool.idle_timeout = 0

This will make your deployments slower, but it might solve the connection errors.

Answer 2

You could try tweaking MaxStartups in /etc/ssh/sshd_config on the server of your Git server.

Default value is 10:30:60, the first 10 limits this.

MaxStartups Specifies the maximum number of concurrent unauthenticated con- nections to the sshd daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10.

    Alternatively, random early drop can be enabled by specifying the
    three colon separated values "start:rate:full" (e.g.,
    "10:30:60").  sshd will refuse connection attempts with a proba-
    bility of "rate/100" (30%) if there are currently "start" (10)
    unauthenticated connections.  The probability increases linearly
    and all connection attempts are refused if the number of unau-
    thenticated connections reaches "full" (60).