sidekiq with multiple queues: jobs get stuck

Question

I have recently had to prioritize my Sidekiq queues (before I was using only the default queue).

So I have written few jobs which are supposed to be executed on the different queues.

My sidekiq.yml :

:queues:
  - slack_notifier
  - invoicing
  - default

deploy.rb (capistrano):

# ...
set :sidekiq_config, "#{current_path}/config/sidekiq.yml"
# probably below line does same thing as above one
set :sidekiq_options_per_process, ['--queue slack_notifier --queue invoicing --queue default']
# ...

And I can see while deploying, that it's being used:

01:32 sidekiq:start
      01 $HOME/.rbenv/bin/rbenv exec bundle exec sidekiq --index 0 --pidfile /home/ubuntu/cap/shared/tmp/pids/sidekiq.pid --environment production --logfile /home/ubuntu/cap/shared/log/sidekiq.log --config /home/ubuntu/cap/current/config/sidekiq.yml --queue slack_notifier --queue invoicing --queue default --daemon

Sidekiq is running:

ps aux | grep '[s]idekiq'
#=> ubuntu     716 16.4  2.1 668604 176860 ? Sl   07:31  14:18 sidekiq 4.2.6 cap [1 of 1 busy]

Now, when I start off the jobs, they just seem to be halted at some point.

Not sure if I can describe the issue better. When I clear out the two priority queues ( slack_notifier and invoicing ) the default still does not seem to process jobs.

Answer 1

Maybe you can run multiple processes for all queues, put separate queues in separate processes for processing sidekiq jobs. Your configuration will look like this

set :sidekiq_processes, 3
set :sidekiq_options_per_process, ['--queue slack_notifier', '--queue invoicing', '--queue default']

I think it will not halt the sidekiq, but the precondition to everything is your sidekiq processes are optimized enough so they are not taking more memory than they are supposed to be using.