rails resque worker memory consumption

Question

I am using a resque gem to process my background process.

I have setup three queue with one worker each.

Can any one explain how the memory gets consumed with increase and decrease of worker in the queue.

I have heard that each worker loads up separate Rails environment. Is that true?

Answer 1

As far as I know, yes, each worker start a separate Rails environment, so if you have three workers you will have three Rails environments loaded. Having more workers won't make your queue to grow longer, it will be the number of jobs queued what will make it to grow.

Either way, if you don't have any major reason to have 3 separate workers I suggest you to have only one worker for all queues and separate them as your application scales with time.

Answer 2

It's not true. As I explained in my comment here, the precise goal of Resque is not to load a rails environment for each worker (see https://github.com/blog/542-introducing-resque ).

Due to a limitation caused by ruby's green trheading solution, you should launch at least one worker per processor core to be able to use the the whole CPU. This is why the default behavior of Resque is to launch each worker in a separate process. It means launch N parallel processes at the same time, each with the full gem set loaded independently. This is the main cause of high memory use for Resque and to any other Ruby tool. You can see here how things can get better using native threading with the JVM.

So if you want keep your Resque memory usage down keep your workers dependencies as low as you can. And always is very important to use a monitoring tool like God or Bluepill to keep an eye in the process.