Memory consumpton inside sidekiq worker

Question

1. Does loading multiple Models in sidekiq worker can cause memory leak? Does it get garbage collected?

For example:

class Worker
  include Sidekiq::Worker

  def perform
    Model.find_each do |item|

    end
  end
end

2. Does using ActiveRecord::Base.connection inside worker can cause problems? Or this connection automatically closes?

Answer 1

I think you are running into a problem that I also had with a "worker" - the actual problem was the code, not Sidekiq in any way, shape or form.

In my problematic code, I thoughtlessly just loaded up a boatload of models with a big, fat, greedy query (hundreds of thousands of instances).

I fixed my worker/code quite simply. For my instance, I transitioned my DB call from all to use find_in_batches with a lower number of objects pulled for the batch.

Model.find_in_batches(100) do |record| 
# ... I like find_in_batches better than find_each because you can use lower numbers for the batch size
# ... other programming stuff 

As soon as I did this, a job that would bring down Sidekiq after a while (running out of memory on the box) has run with find_in_batches for 5 months without me even having to restart Sidekiq ... Ok, I may have restarted Sidekiq some in the last 5 months when I've deployed or done maintenance :), but not because of the worker!