分布式环境中的任务执行

Question

I have a table say employee(id, name, email, is_email_verified, created_date). I need to send an email everyday for the users which has been created and email is not verified. I need to create a service which can be deployed on two boxes. It will read the user table and get the user id and send the mail for verification. Since, there are two boxes and each will read the same number of users and hence will send the two mails.

How, can we make this service so as user ids will get distributed among the servers (if I have n servers).

Answer 1

There are few ways to do that.

The most straightforward & reliable:

1. Create a job task that read all user ID from the table and queue them to a task queue - 1 task per user ID.
2. Pull task from queue using as many servers as you want (1-N).

Not requiring queues (not fault tolerant):

Choose a hash function that would decide which user IDs should server process. For example 1st server processes odd IDs and 2nd even IDs, etc. Or something like:

def should_i_send(user_id, server_id):
  return user_id % server_id == 0