Rails 延迟作业和 docker：添加更多工人

Question

I run my rails app using a Docker. Delayed jobs are processed by a single worker that runs in a separate container called worker and inside it worker runs with a command bundle exec rake jobs:work .

I have several types of jobs that I would like to move to a separate queue and create a separate worker for that. Or at least have two workers for process tasks.

I tried to run my worker container with env QUEUE=default_queue bundle exec rake job:work && env QUEUE=another_queue bundle exec rake job:work but that does not make any sense. It does not fails, is starts but jobs aren't processed.

Is there any way to have separate workers in one container? And is it correct? Or should I create separate container for each worker I would ever want to make?

Thanx in advance!

Answer 1

Running the command command1 && command2 results in command2 being executed only when command1 completes. rake jobs:work never terminates, even when it has finished executing all the jobs in the queue, so the second command will never execute.

A single "&" is probably what you're looking for: command1 & command2 . This will run the commands independently in their own processes.

You should use the delayed_job script on production, and it's a good idea to put workers of different queues into different containers in case one of the queues contains jobs that use up a lot of resources.

This will start a delayed job worker for the default_queue:
bundle exec script/delayed_job --queue=default_queue start
For Rails 4, it is: bundle exec bin/delayed_job --queue=default_queue start
Check out this answer on the topic: https://stackoverflow.com/a/6814591/6006050

You can also start multiple workers in separate processes using the -n option. This will start 3 workers in separate processes, all picking jobs from the default_queue:
bundle exec script/delayed_job --queue=default_queue -n 3 start

Differences between rake jobs:work and the delayed_job script:
It appears that the only difference is that rake jobs:work starts processing jobs in the foreground, while the delayed_job script creates a daemon which processes jobs in the background. You can use whichever is more suited to your use case.
Check this github issue: https://github.com/collectiveidea/delayed_job/issues/659

Answer 2

Actually i just came across this problem with scaling delayed_jobs on docker

see this gist for a script that starts delayed jobs with arbitrary arguments and listens to SIGTERM and executes a smooth shutdown of the started jobs on container shutdown. This way you can execute as many processes and queues as you want.

https://gist.github.com/jklimke/3fea1e5e7dd7cd8003de7500508364df

#!/bin/bash

# Variable DELAYED_JOB_ARGS contains the arguments for delayed jobs for, e.g. defining queues and worker pools.

# function that is called when the docker container should stop. It stops the delayed job processes
_term() {
  echo "Caught SIGTERM signal! Stopping delayed jobs !"
  # unbind traps
  trap - SIGTERM
  trap - TERM
  trap - SIGINT
  trap - INT
  # end delayed jobs 
  bundle exec "./bin/delayed_job ${DELAYED_JOB_ARGS} stop"

  exit
}

# register handler for selected signals
trap _term SIGTERM
trap _term TERM
trap _term INT
trap _term SIGINT

echo "Starting delayed jobs ... with ARGs \"${DELAYED_JOB_ARGS}\""

# restart delayed jobs on script execution
bundle exec "./bin/delayed_job ${DELAYED_JOB_ARGS} restart"

echo "Finished starting delayed jobs... Waiting for SIGTERM / CTRL C"

# sleep forever until exit
while true; do sleep 86400; done