Can Resque run jobs in parallel?

Question

Can Resque run a number of similar jobs all in parallel? If so, how would you set up workers to do this?

Answer 1

I'd take a look at the Resque Pool gem , it can make it really easy to set up multiple workers for different queues. Then you can group similar tasks by queue.

If you aren't already using something for process monitoring, resque-pool can help you manage your works via a single deamon. It also gives you a few ways to be able to monitor what is going on with your workers. Also it lets you easily change how many workers are running for each queue through a simple yaml file.

Answer 2

If you pass a COUNT option to the rake task, you can run multiple workers:

COUNT=5 QUEUE=* rake resque:workers

In production, you could use something like God to keep your processes running. The Resque project has a sample configuration script which allows for multiple workers.

Answer 3

If you monitoring with monit be careful have one pid file per worker

#!/bin/sh
cd <path of app>/current/
for i in 3; 
do
    VVERBOSE=1 PIDFILE=<path of app>/shared/pids/resque_$i.pid RAILS_ENV=production QUEUE=name_of_queue bundle exec rake environment resque:work > <path of app>/shared/log/resque_$i.log
done