Sidekiq threads accessing global variable

Question

I have a controller that spins off 6 sidekiq threads for faster parallel processing of a large file. Before that however I want to provide these threads with a few variables that should be available accross all threads because they variables themselves are fairly memory intensive. (it is only reading from that, not writing, so the concurrency issues doesn't exist)

In other words my controller looks like this

def foo
  $bar1 = ....
  $bar2 = ...
  worker.perform_async()...
  worker2.perform_async()...
end

I don't want to put those global vars into the perform methods because serializing those to redis chokes the entire thing. My issue is that the workers cannot see these variables and die because of a no method error (ie trying to call .first on on of them gives that error because the var is nil for the workers).

How come? Is there any other way to do this that won't kill my memory? (ie I don't want to take up most of the mem with 6x the same large array)

Answer 1

Sidekiq runs on a separate process , so it doesn't share the same memory as the initiator of the worker.

If the data is static, you might want to load it on the start of the sidekiq process (maybe when you configure the sidekiq server).

If it changes per task, you should model it in a way where you can create a global repository to hold it (if redis is not good for this, maybe you can try memcached )...