How to shuffle jobs in a Resque queue?

Question

I have a queue named check_integrity and lots of jobs in it. When i run a worker for it it takes jobs first in first out order. Is it possible to shuffle the jobs in that particular queue? I need the worker to take jobs randomly. Please help.

Thanks.

Answer 1

One way to go about this is by popping entries out of the queue, batching them up, shuffling the batch and then re-insert them:

key = "resque:queue:bulk"
total = Redis.current.llen(key)
batch_size = 5_000 # any value that is good enough for you

batch = []
total.times do |i|
  entry = Redis.current.lpop(key)
  batch << entry
  if batch.size == batch_size || i.succ == total
    puts "re-inserting batch..."
    Redis.current.rpush key, batch.shuffle
    batch = []
  end
end

This is really useful when you mistakenly enqueue several jobs that end up racing for shared resources, locks and so on.

Answer 2

Look at this plugin for the Resque. I guess this is exactly what you need.

Answer 3

If you don't mind monkey patching resque then you can use this solution:

module Resque

  # Monkey patch Resque to handle queues as sets instead of lists. This allows
  # use to get jobs randomly rather then sequentially.
  def push(queue, item)
   watch_queue(queue)
   redis.sadd "queue:#{queue}", encode(item)
  end

  def pop(queue)
    decode redis.spop("queue:#{queue}")
  end

  def size(queue)
    redis.scard("queue:#{queue}").to_i
  end
end

If are using rails create a file inside the initializers with that code and you will be set.

Answer 4

you can use Delayed_job