Retry Sidekiq worker from within worker

Question

In my app I am trying to perform two worker tasks sequentially. First, a PDF is being created with Wicked pdf and then, once the PDF is created, to send an email to two different recipients with the PDF attached.

This is what is called in the controller :

PdfWorker.perform_async(@d.id)
MailingWorker.perform_in(1.minutes, @d.id,@d.class.name.to_s)

First worker creates the PDF and second worker sends email.

Here is second worker :

class MailingWorker
  include Sidekiq::Worker
  sidekiq_options retry: false    
  def perform(d_id,model)

    @d = eval(model).find(d_id)
    @model = model
    if @d.pdf.present?
      ProfessionnelMailer.notification_d(@d).deliver
      ClientMailer.notification_d(@d).deliver
    else
      MailingWorker.perform_in(1.minutes, @d.id, @model.to_s)
    end
  end
end

The if statement checks if the PDF has been created. If true two mails are sent, otherwise, the same worker is called again one minute later, just to let the Heroku server extra time to process the PDF creation in case it takes more time or a long queue.

Though if the PDF has definitely failed to be processed, the above ends up in an infinite loop.

Is there a way to fix this ?

One option I see is calling the second worker inside the PDF creation worker though I don't really want to nest workers too deep. It makes my controller more clear to have them separate, I can see the sequence of actions. But any advice welcome.

Another option is to use sidekiq_options retry: 5 and request a retry of the controller that could be counted towards the full total of 5 retries, instead of retrying the worker with else MailingWorker.perform_in(1.minutes, @d.id, @model.to_s) but I don't know how to do this. As per this thread https://github.com/mperham/sidekiq/issues/769 it would be to raise an exception but I am not sure how to do this ... (also I am not sure how long the retry will wait before being processed with the exception method, with the solution above I can control the time frame..)

Answer 1

I believe the "correct" and most asynchroneous way to do this is to have two queues, and two workers:

• Queue 1: CreatePdfWorker
• Queue 2: SendPdfWorker

When the CreatePdfWorker has generated the PDF, it then enqueues the SendPdfWorker with the newly generated PDF and recipients.

This way, each worker can work independently and pluck from the queue asynchroneously, and you're not struggling against the design choices of Sidekiq.

Answer 2

If you do not want to have nested workers, then in MailingWorker instead of enqueuing it again, raise an exception if the PDF is not present. Also, configure the worker retry option, so that sidekiq will push it to the retry queue and run it again in sometime. According to the documentation,

Sidekiq will retry failures with an exponential backoff using the 
formula (retry_count ** 4) + 15 + (rand(30) * (retry_count + 1)) (i.e. 
15, 16, 31, 96, 271, ... seconds + a random amount of time). It will 
perform 25 retries over approximately 21 days.

Worker code will be more like,

class MailingWorker
  include Sidekiq::Worker
  sidekiq_options retry: 5

  def perform(d_id,model)

    @d = eval(model).find(d_id)
    @model = model
    if @d.pdf.present?
      ProfessionnelMailer.notification_d(@d).deliver
      ClientMailer.notification_d(@d).deliver
    else
      raise "PDF not present"
    end
  end
end