Ruby on Rails Sidekiq order of workers execution

Question

I have rails app with Sidekiq background workers

MyWorker1.perform_async(param_1)
MyWorker1.perform_async(param_2) 
MyWorker1.perform_async(param_3)
MyWorker1.perform_async(param_4)

MyWorker2.perform_async(param_5) 

How to make MyWorker2 to be executed only if all MyWorker1 finished work?

Answer 1

you can use GUSh

class SimpleworkFlow < Gush::Workflow
   run MyWorker1, params: {param_1}
   run Myworker1, params: {param_2}
   run MyWorker1, params: {param_3}
   run MyWorker1, params: {param_4}
   run Myworker2, params: {param_5}, after: MyWorker1
end

and for debugging purpose you can visualize

bundle exec gush viz SampleWorkflow

then simple steps

flow = SampleWorkflow.new
flow.save 

Then start worker

bundle exec gush workers

Then start workflow

flow.start!

Answer 2

Either you pay for it and use Sidekiq pro with Batches :

class CallNextJob
  def on_success(status, options)
    MyWorker2.perform_async(options[:bundle])
  end
end

params = [param1, param2, param3, param4]
batch = Sidekiq::Batch.new
batch.on(:success, CallNextJob, bundle: param5)
batch.jobs do
  params.each {|param| MyWorker1.perform_async(param)}
end
puts "Just started Batch #{batch.bid}"

Or you just save completion status at the end of each job, and enqueue MyWorker2 if every MyWorker1 is finished:

# Worker1
def heavy_load(job_id)
  # do the load
  ...
  # save state
  WorkerReport.create(worker_id: job_id)
end

class WorkerReport < ActiveRecord::Base
  after_commit :do_next

  private

  # It's important to do that in after_commit block to evade strange bugs
  # When you create next job here, you are absolutely sure that
  # current WorkerReport is already saved to base
  def do_next
    # Check if other Worker1 jobs from current bundle are finished
    if self.where(...).exists?
      Worker2.perform_async(...)
    end
  end
end

Answer 3

Maybe you can persist the result of your Worker1 result and check at the end if your result is complete. Than start worker2 from your worker1 if you have a complete result?

MyWorker1.perform_async(param_1) -> {done: false},

MyWorker1.perform_async(param_2) -> {done: false},

MyWorker1.perform_async(param_3) -> {done: false},

MyWorker1.perform_async(param_4) -> {done: true},

MyWorker1 -> if result.done? MyWorker2