Rails' DelayedJob for a multi-master setup

Question

I want to add one or two servers to my setup. I want it to be a multi-master setup with Postgresql-BDR to be available even if one server crashes.

Now the problem: I use delayed_job pretty heavily. A job should only be run on one server to avoid problems with APIs. I could run the jobs only on one machine, but everything will stop working when that machine crashes.

Does anyone has a nice solution for this case?

Answer 1

I have a solution for this case. I use a custom worker that checks if the job is already running on another server. If it is, it will not run the job. If it is not, it will run the job and set a lock on the job. The lock will be removed when the job is finished.