How to monitor delayed_job with monit

Question

Are there any examples on the web of how to monitor delayed_job with Monit?

Everything I can find uses God, but I refuse to use God since long running processes in

Answer 1

Since i didn't want to run as root, I ended up creating a bash init script that monit used for starting and stopping (PROGNAME would be the absolute path to script/delayed_job):

start() {
    echo "Starting $PROGNAME"
    sudo -u $USER /usr/bin/env HOME=$HOME RAILS_ENV=$RAILS_ENV $PROGNAME start
}

stop() {
    echo "Stopping $PROGNAME"
    sudo -u $USER /usr/bin/env HOME=$HOME RAILS_ENV=$RAILS_ENV $PROGNAME stop
}

Answer 2

I have spent quite a bit of time on this topic. I was fed up with not having a good solution for it so I wrote the delayed_job_tracer plugin that specifically addresses monitoring of delayed_job and its jobs.

Here's is an article I've written about it: http://modernagility.com/articles/5-monitoring-delayed_job-and-its-jobs

This plugin will monitor your delayed job process and send you an e-mail if delayed_job crashes or if one of its jobs fail.

Answer 3

to see what is going on, run monit in foreground verbose mode: sudo monit -Iv

using rvm installed under user "www1" and group "www1".

in file /etc/monit/monitrc:

#delayed_job
check process delayed_job with pidfile /home/www1/your_app/current/tmp/pids/delayed_job.pid
    start program "/bin/bash -c 'PATH=$PATH:/home/www1/.rvm/bin;source /home/www1/.rvm/scripts/rvm;cd /home/www1/your_app/current;RAILS_ENV=production bundle exec script/delayed_job start'" as uid www1 and gid www1
    stop program "/bin/bash -c 'PATH=$PATH:/home/www1/.rvm/bin;source /home/www1/.rvm/scripts/rvm;cd /home/www1/your_app/current;RAILS_ENV=production bundle exec script/delayed_job stop'" as uid www1 and gid www1
    if totalmem is greater than 200 MB for 2 cycles then alert

Answer 4

Thanks for the script.

One gotcha -- since monit by definition has a 'spartan path' of

/bin:/usr/bin:/sbin:/usr/sbin

... and for me ruby was installed / linked in /usr/local/bin, I had to thrash around for hours trying to figure out why monit was silently failing when trying to restart delayed_job (even with -v for monit verbose mode).

In the end I had to do this:

check process delayed_job with pidfile /var/www/app/shared/pids/delayed_job.pid
start program = "/usr/bin/env PATH=$PATH:/usr/local/bin /var/www/app/current/script/delayed_job -e production start"
stop program = "/usr/bin/env PATH=$PATH:/usr/local/bin /var/www/app/current/script/delayed_job -e production stop"

Answer 5

For Rails 3, you may need set HOME env to make compass work properly, and below config works for me:

check process delayed_job
  with pidfile /home/user/app/shared/pids/delayed_job.pid
  start program = "/bin/sh -c 'cd /home/user/app/current; HOME=/home/user RAILS_ENV=production script/delayed_job start'"
  stop program  = "/bin/sh -c 'cd /home/user/app/current; HOME=/home/user RAILS_ENV=production script/delayed_job stop'"

Answer 6

If your monit is running as root and you want to run delayed_job as my_user then do this:

/etc/init.d/delayed_job:

#!/bin/sh
#   chmod 755 /etc/init.d/delayed_job
#   chown root:root /etc/init.d/delayed_job

case "$1" in
  start|stop|restart)
    DJ_CMD=$1
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit
esac

su -c "cd /var/www/my_app/current && /usr/bin/env bin/delayed_job $DJ_CMD" - my_user

/var/www/my_app/shared/monit/delayed_job.monitrc:

check process delayed_job with pidfile /var/www/my_app/shared/tmp/pids/delayed_job.pid
start program = "/etc/init.d/delayed_job start"
stop  program = "/etc/init.d/delayed_job stop"
if 5 restarts within 5 cycles then timeout

/etc/monit/monitrc:

# add at bottom
include /var/www/my_app/shared/monit/*