Rails deployment on VPS with Capistrano, Unicorn and RVM

Question

Upon

cap deploy:cold

my capistrano recipe works great but gets stuck on this part:

  * 2013-01-06 23:07:08 executing `deploy:start'
  * executing "/etc/init.d/seventysix_unicorn start"
    servers: ["xxxxxxxxxxxxx"]
    [xxxxxxxxxxxxx] executing command
  ** [out :: xxxxxxxxxxxxx] Password:

And waits here forever. Upon visiting the webpage I can see that nginx is running properly, but unicorn isn't. Even when I start unicorn manually by doing:

cd $APP_ROOT; bundle exec unicorn -c $APP_ROOT/config/unicorn.rb

and reload the page, the page seems to be processing something (that circle icon indicating progress in a browser is spinning) but after 30secs or so, it stops, rerouting me to default 500.html.

Here are my deployment configs.

nginx.conf

  1 upstream unicorn {
  2   server unix:/tmp/unicorn.seventysix.sock fail_timeout=0;
  3 }
  4 
  5 server {
  6 
  7   listen 80 default deferred;
  8   server_name www.example.com example.com;
  9   root /home/mr_deployer/apps/seventysix/current/public;
 11 
 12   location ^~ /assets/ {
 13     gzip_static on;
 14     expires max;
 15     add_header Cache-Control public;
 16   }
 17 
 18   try_files $uri/public $uri @unicorn;
 19 
 20   location @unicorn {
 21     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 22     proxy_set_header Host $http_host;
 23     proxy_redirect off;
 24     proxy_pass http://unicorn;
 26   }
 27 
 28   error_page 500 502 503 504 /500.html;
 29   client_max_body_size 4G;
 30   keepalive_timeout 10;
 31 
 32 }

unicorn.rb

  1 root = "/home/mr_deployer/apps/seventysix/current"
  3 
  4 working_directory root
  5 pid "#{root}/tmp/pids/unicorn.pid"
  6 stderr_path "#{root}/log/unicorn.log"
  7 stdout_path "#{root}/log/unicorn.log"
  8 
  9 listen "/tmp/unicorn.seventysix.sock"
 10 worker_processes 2
 11 timeout 30

unicorn_init.sh

  1 #!/bin/sh

 28 APP_ROOT=/home/mr_deployer/apps/seventysix/current
 29 RAILS_ENV=production
 32 USER=mr_deployer
 33 
 34 PID=$APP_ROOT/tmp/pids/unicorn.pid
 35 CMD="cd $APP_ROOT; bundle exec unicorn -c $APP_ROOT/config/unicorn.rb -E $RAILS_ENV -D"
 38 
 39 action="$1"
 40 set -u
 41 
 42 old_pid="$PID.oldbin"
 43 
 44 cd $APP_ROOT || exit 1
 45 
 46 sig () {
 47   test -s "$PID" && kill -$1 `cat $PID`
 48 }
 49 
 50 oldsig () {
 51   test -s $old_pid && kill -$1 `cat $old_pid`
 52 }
 53 
 54 case $action in
 55 
 56 start)
 57   sig 0 && echo >&2 "Already running" && exit 0
 58   su $USER -c "$CMD"
 59   ;;
 60 
 61 stop)
 62   sig QUIT && exit 0
 63   echo >&2 "Not running"
 64   ;;
 65 
 66 force-stop)
 67   sig TERM && exit 0
 68   echo >&2 "Not running"
 69   ;;
 70 
 71 restart|reload)
 72   sig HUP && echo reloaded OK && exit 0
 73   echo >&2 "Couldn't reload, starting '$CMD' instead"
 74   su $USER -c "$CMD"
 75   ;;
 76 
 77 upgrade)
 78   if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
 79   then
 80     n=$TIMEOUT
 81     while test -s $old_pid && test $n -ge 0
 82     do
 83       printf '.' && sleep 1 && n=$(( $n - 1 ))
 84     done
 85     echo
 86     if test $n -lt 0 && test -s $old_pid
 87     then
 88       echo >&2 "$old_pid still exists after $TIMEOUT seconds"
 89     exit 1
 90   fi
 91   exit 0
 92   fi
 93   echo >&2 "Couldn't upgrade, starting '$CMD' instead"
 94   su $USER -c "$CMD"
 95   ;;
 96 reopen-logs)
 97   sig USR1
 98   ;;
 99 
100 *)
101 
102 echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
103   exit 1
104   ;;
105 esac

deploy.rb

  1 require "bundler/capistrano"
  2 
  3 set :rvm_ruby_string, '1.9.3-p125@seventysix'
  4 
  5 require "rvm/capistrano" # Load RVM's capistrano plugin.
  6 
  7 server "176.58.126.11", :web, :app, :db, primary: true
  8 
  9 set :application, "seventysix"
 10 set :user, "mr_deployer"
 11 set :deploy_to, "/home/#{user}/apps/#{application}"
 12 set :deploy_via, :remote_cache
 13 set :use_sudo, false
 14 
 15 set :scm, "git"
 16 set :repository, "git@github.com:ofcan/#{application}.git"
 17 set :branch, "master"
 18 
 19 default_run_options[:pty] = true
 20 ssh_options[:forward_agent] = true
 21 
 22 after "deploy", "deploy:cleanup" # keep only the last 5 releases
 23 
 24 namespace :deploy do
 25 
 26   %w[start stop restart].each do |command|
 27     desc "#{command} unicorn server"
 28       task command, roles: :app, except: {no_release: true} do
 29       run "/etc/init.d/#{application}_unicorn #{command}"
 30     end
 31   end
 32 
 33   task :setup_config, roles: :app do
 34     sudo "ln -nfs #{current_path}/config/nginx.conf /etc/nginx/sites-enabled/#{application}"
 35     sudo "ln -nfs #{current_path}/config/unicorn_init.sh /etc/init.d/#{application}_unicorn"
 36     run "mkdir -p #{shared_path}/config"
 37     put File.read("config/database.example.yml"), "#{shared_path}/config/database.yml"
 38     puts "Now edit the config files in #{shared_path}."
 39   end
 40 
 41   after "deploy:setup", "deploy:setup_config"
 42 
 43   task :symlink_config, roles: :app do
 44     run "ln -nfs #{shared_path}/config/database.yml #{release_path}/config/database.yml"
 45   end
 46 
 47   after "deploy:finalize_update", "deploy:symlink_config"
 48     desc "Make sure local git is in sync with remote."
 49     task :check_revision, roles: :web do
 50       unless `git rev-parse HEAD` == `git rev-parse github/master`
 51       puts "WARNING: HEAD is not the same as github/master"
 52       puts "Run `git push` to sync changes."
 53       exit
 54     end
 55   end
 56 
 57   before "deploy", "deploy:check_revision"
 58 
 59 end

I am also using RVM. I know the RVM docs suggest creating wrapper for unicorn, but I cant figure out how exactly to create it. Also, why is Unicorn, when manually started on VPS with that command I said earlyer, timing out?

Answer 1

I think that your unicorn start command is running as sudo and needing a password (sudoless password isn't setup).

I'm basing this on the the assumption that the rest of your capistrano script has executed correctly and that it has connected to the remote server to do so (therefore this isn't SSH requesting interactive login).

and reload the page, the page seems to be processing something (that circle icon indicating progress in a browser is spinning) but after 30secs or so, it stops, rerouting me to default 500.html.

This is nginx trying to connect to the downstream (unicorn) server and timing out after the default timeout (30s), and returning a 500.

This is odd, as it implies that the unix socket is active (otherwise nginx would return bad gateway, not server error), but the socket isn't responding.

Answer 2

I had a similar problem this week, with Nginx 1.2.6.

I'm going to shoot all the solutions that may have fixed that, I spent plenty of time to make it work. Comment if something worked for you:

1. I removed gzip (the entire location ^~ /assets/ node). I don't know why it didn't work, and I'm hosting assets on S3 so I didn't care much.

2. To access unicorn from the bash script you have two options: with bundle or with a booted executable:

using unicorn with bundle :

You need to auto trust RVM in your APP_ROOT, by creating a file in your home directory: /home/mr_deployer/.rvmrc with the content:

export rvm_trust_rvmrcs_flag=1

Then, if you don't have it yet, add .rvmrc file to your APP_ROOT with

rvm --create use  "1.9.3-p125@seventysix"

And finally change the ; into && :

cd $APP_ROOT && bundle exec unicorn -c $APP_ROOT/config/unicorn.rb

or, using unicorn with rvm booted executable :

Run this on your server:

rvm wrapper 1.9.3-p125@seventysix bootup unicorn

Then, your command should look like

 /home/mr_deployer/.rvm/bin/bootup_unicorn -c $APP_ROOT/config/unicorn.rb -E $RAILS_ENV -D

I should add that I'm running service unicorn_blah from my deployer user and it works fine. No root or sudo.