How to monitor nginx passenger with monit

Question

I have several rails applications deployed by nginx passenger. I want those applications to be monitored by using monit. How can I monitor those applications using monit? Should I monitor nginx as well?

Answer 1

This is how I solved this. First, I added to application.rb:

# Monit support
if defined?(PhusionPassenger)
  require 'pidfile_manager'
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    if forked
      # We're in smart spawning mode.
      PidfileManager.write_pid_file
    else
      # We're in conservative spawning mode. We don't need to do anything.
    end
  end

  PhusionPassenger.on_event(:stopping_worker_process) do
    PidfileManager.remove_pid_file
  end
end

and then I implemented the PidfileManager:

module PidfileManager
  extend self

  BASENAME = '/var/tmp/rack.*.pid'

  def write_pid_file
    pid = Process.pid
    count = 1
    pidfile = nil
    go_over_pid_files do |file, saved_pid|
      file_id = file[/(\d+)/,1].to_i
      # Increase counter only if we met the same file id
      count += 1 if file_id == count
      # We're already there
      return if saved_pid == pid
      # Check if the process is alive
      res = begin
        Process.kill(0, saved_pid)
      rescue Errno::ESRCH
        nil
      end
      # It's dead, reuse
      unless res
        pidfile = file
        break
      end
    end
    pidfile ||= BASENAME.sub('*', count.to_s)
    File.open(pidfile, 'w') {|f| f.write(pid.to_s)}
  end

  def remove_pid_file
    pid = Process.pid
    go_over_pid_files do |file, saved_pid|
      if pid == saved_pid
        File.unlink(file)
        break
      end
    end
  end

  private
  def go_over_pid_files
    Dir[BASENAME].each do |file|
      saved_pid = File.read(file).to_i
      yield file, saved_pid
    end
  end

end

And then you just tell monit to monitor each instance using /var/tmp/rack.X.pid as a pidfile.

Answer 2

Not sure if its too late to post this but this is how I'm using monit (5.14) to stop passenger rails apps consuming too much memory:

monit:

check program ourapp_live with path     "/usr/local/bin/check_passenger_mem_usage ourapp 500" as "ourappuser"
 if status != 0 then alert
 if status != 0 then restart
 start program = "/bin/touch /var/www/vhosts/ourapp/railsapp/current/tmp/restart.txt"
 stop program = "/bin/true"

The shell script monit calls (check_passenger_mem_usage):

#!/bin/bash
#
#
USER=$1
TRIGGER=$2

if [ -z $USER ] || [ -z $TRIGGER ]
then
    echo "missing args"
    echo "usage:"
    echo "         check_passenger_mem_usage username alert_threshold"
    echo
    echo "(alert_threshold is in mb)"
    echo
    exit 1
fi

MAX=`/usr/local/rvm/gems/ruby-1.8.7-p357/wrappers/passenger-memory-stats | grep $USER | awk '{print $2}' | sort -n | tail -1|cut -d. -f1`

if [ "$MAX" -gt $TRIGGER ]
then
    echo
    echo "${USER}: We got a runaway! Gobbling ${MAX} mb"
    echo
    exit 1
else
    echo
    echo "Max is ${MAX}"
    exit 0
fi

Probably not the best solution as it restarts the entire rails app but at least it prevents rails consuming vast amounts of memory should the app go a little memory intensive now and again.

Answer 3

If you're looking to keep them running and have them restart if there's an error, you may be better looking at supervisord . Rather than polling to see whether a process is running, supervisord actually runs the process itself. The daemons it runs need to run the the foreground for it to work, but its very effective and will bring services up much quicker than monit would (monit usually polls every minute, whereas supervisord would see the process end and restart it instantly).

We use supervisord in production running all our daemons (nginx, beanstalkd, memcached, various python services, etc), and then use monit to monitor supervisord as an added back-up.