使用数据库作为任务队列

Question

In one of our java applications (based on postgresql db), we have a database table that maintains a list of tasks to be executed. Each row has a json blob for the details of a task as well as scheduled time value.

We have a few java workers/threads whose jobs are to search for tasks that are ready for execution (based on its schedule value), execute and delete them from the table. Execution of a task may take a few seconds.

The problem is, more than one worker may grab the same row, causing duplicate execution of a task, which is something we want to avoid.

One approach is, when doing select to grab a row, do it with FOR UPDATE to lock the row, supposedly preventing other worker from grabbing the same row that's locked.

My concern with this approach is, the row is only locked when the select transaction is being executed in the db (according to this ), while the java code is actually executing the row/task that's selected, the locking has gone, another worker can grab it again.

Can some shed some light on whether the above approach is going to work for sure? Thanks!

Answer 1

Treat the DB calls as atomic instructions and design lock free algos around your table, using updates to change a boolean column "in-progress" from false to true. Could also just be a state int (0=avail, 1=inprogress, N=resultcode).

Make sure you have a partial index on state 0 (and possibly 1 to recover from crashes to find tasks in progress), so that the ...where state=0 remains selective and fast (on top of the scheduled time index of course).

Hope this helps.

Answer 2

When one thread has successfully locked the row on a given connection, another one attempting to obtain a lock on the row on a different connection should fail. You should issue the select-for-update with some kind of no-wait clause to request immediate failure if the row is locked.

Now, this doesn't solve the query vs lock race, as a failed lock may interrupt a thread's execution. You can solve that by (in each execution):

1. Select all records with new tasks (regardless of whether they're being processed or not)
2. For each new task returned in [1], run a matching select-for-update, then continue with processing the task if the lock fails.
3. If any lock attempt fails, skip the task without failing the entire process.