How can I get notified when Sidekiq Enqueues jobs or has stopped processing jobs?

Question

I am on Heroku and I got an error because my redis db got too full. The my sidekiq processes stopped working. It was like that for a day until I realized it. Now I have 600+ jobs that I have tried to process but they are just breaking everything now. How can I sound off the alarms when sidekiq can't process jobs or when the Enqueue starts to fill up?

Answer 1

You could set a rake task on a schedule to check Sidekiq stats, and then take the appropriate action ( like send an email ).

I've created my own module with helper methods for Sidekiq that serves a number of purposes, eg deleting jobs, checking queues, retriving jobs by certain criteria etc. https://gist.github.com/blotto/10324119

For your purpose, grab the sidekiq stats as such :

 def sidekiq_stats()
   summary = Hash.new
   stats = Sidekiq::Stats.new
   summary = { processed: stats.processed,
              failed: stats.failed,
              enqueued: stats.enqueued,
              queues: stats.queues}
 end

And then evaluate the enqueued value, set a tolerance on what you think is too high, and then let loose the hounds.

Answer 2

If you're using zabbix for monitoring, you could use sidekiq_queue_zabbix template at https://github.com/hungntit/sidekiq_queue_zabbix . This template supports showing graph and sending alert when sidekiq queue size is higher than one specified limit number