monit and unicorn restart command not working in rails 5.2 application

Question

I configured Monit to watch unicorn and restart it when the memory exceeded or the cpu increased above a certain limit ,

how ever when it happens , monit doesn't restart unicorn , and here is the logs I found in monit log file

[UTC Aug 11 20:15:41] error    : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process

Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current;  bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:16:11] error    : 'unicorn_myapp' process is not running
[UTC Aug 11 20:16:11] info     : 'unicorn_myapp' trying to restart
[UTC Aug 11 20:16:11] info     : 'unicorn_myapp' restart: '/etc/init.d/unicorn_myapp restart'
[UTC Aug 11 20:16:42] error    : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process

Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current;  bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:17:12] error    : 'unicorn_myapp' process is not running
[UTC Aug 11 20:17:12] info     : 'unicorn_myapp' trying to restart
[UTC Aug 11 20:17:12] info     : 'unicorn_myapp' restart: '/etc/init.d/unicorn_myapp restart'
[UTC Aug 11 20:17:42] error    : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process

Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current;  bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:18:12] error    : 'unicorn_myapp' process is not running

Here is my monit configuration under /etc/monit/conf.d/

check process unicorn_myapp
  with pidfile /home/ubuntu_user/apps/myapp/current/tmp/pids/unicorn.pid
  start program = "/etc/init.d/unicorn_myapp  start"
  stop program = "/etc/init.d/unicorn_myapp  stop"
  restart program = "/etc/init.d/unicorn_myapp  restart"
  if not exist  then restart
  if mem is greater than 300.0 MB for 2 cycles then restart       # eating up memory?
  if cpu is greater than 50% for 4 cycles then restart                  # send an email to admin
  if cpu is greater than 80% for 30 cycles then restart                # hung process?
  group unicorn

it should restart unicorn when such error happens which break the app

from unicorn.log file

ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)

when I run /etc/init.d/unicorn_myapp restart from terminal it works

Answer 1

Monit mostly uses the restart program to start a program. I do not know why this is, but I also observed this behavior.

Just try to comment out the "restart" line. This should force monit to run the start script, what should not try to kill an existing process.

You might also want to watch your log-file like

CHECK FILE unicorn_log PATH log__file__path___change_me_or_the_world_will_flatten
    start program = "/etc/init.d/unicorn_myapp start"
    stop  program = "/etc/init.d/unicorn_myapp stop"

    # No log File entry for one hour? 
    if timestamp is older than 1 hour then restart

    # Allocate Memery error?
    if content = "Cannot allocate memory" then restart