用于电子邮件处理的同一表的 Java 并发查询

Question

I have a MySQL table that acts as an email queue - holding all records that need to be sent. I am trying to perform the sending of each email with multiple threads. Each thread has to make a query to this email queue table to grab a set of records that will then be sent and deleted from the table.

How do you decide what records each thread will grab from the table? From there, how do you manage these concurrent queries? I am using Java Spring Boot with Hibernate.

Answer 1

I would imagine something like below. This is done using id of a row in database. Not really a good solution if you have huge id gaps. You can refactor this to use some date column or any other data that could help to batch records.

10 - number of threads

i - number of current thread we are iterating through

10000 - batch size, used to get batch of 10000 emails

counter - variable to know which thread should be responsible for which id batch

maximumEmailId - maximum id of email in emails table

 1. Create 10 numbered threads - 0, 1, 2..., 9 2. Start every thread 3. For each thread number (i): 4. For counter = 0, step 10000 5. if (counter / 10000) % 10 == i then - SELECT * FROM emails WHERE id BETWEEN (counter) AND (counter + 10000) - Send emails 6. if counter > maximumEmailId then break;

It will behave like this:

 iteration 0: -thread 0 - counter = 0 - select ... where id between 0 and 10000 -thread 1 - counter = 10000 - select ... where id between 10000 and 20000 -thread 9 - counter = 90000 - select ... where id between 90000 and 100000 iteration 1: -thread 0 - counter = 100000 - select ... where id between 100000 and 11000 -thread 1 - counter = 110000 - select ... where id between 110000 and 12000 -thread 9 - counter = 190000 - select ... where id between 190000 and 20000

Basically, this solution has nothing to do with locking, technical concurrent tricks etc., you just divide your dataset so no-one tries to read the same batch. Just imagine 100 boxes on the ground, thread 0 takes boxes numbered 0, 10, 20,...,90 , thread 1 boxes 1, 11, 21,...,91 etc.