jobrunr - spring boot - prevent processing of duplicates in batch

Question

I am using jobrunr to execute some work in my microservices. The idea is that this job runs every 10 seconds it fetches 1000 records from database (that meet certain criteria) - These records are saved in the database regularly (lets say every minute) by a different process. After retrieving 1k records the job processes them individually to execute some business login (includes http calls) and in the end it deletes the record from the database.

These works fine for one replica.

Now when I have +2 replicas of the same services how can I avoid both replicas processing the same records?

There is a way when I get the first 1k records I could apply some lock on them, so that other replicas will not get these same records while they are being processed. But is there a simpler way to execute this with jobrun?

Appreciate your help

Answer 1

How about split the records to 2 parts?

job1 fetch records: select * from tables where id % 2 = 0 limit 1000;

job2 fetch records: select * from tables where id % 2 = 1 limit 1000;