Run an operation only on 1 EC2 even if there are 5 instances

Question

Use-case:

I have to scan a DynamoDB table periodically in 30 secs. I have five EC2 instances running on a fleet, but I want when the 30secs interval is over, only one EC2 instance out of the five should execute the scan in DynamoDB not all.

How can I implement this mechanism where if an operation is about to start only 1 EC2 instance should pick up the operation not all.

Answer 1

To accomplish your goal you will have to come up with a strategy for your EC2 hosts to agree on which one will do the scan.

There are multiple possible solutions but one that is probably fairly simple to implement would be to use optimistic concurrency in DynamoDB to select the EC2 host for each scan.

Either in the same table (if schema permits) or in a separate table create a scan-schedule item with the following attributes:

• key - a primary key (can also be composite if you need to retrofit into your existing table schema) set to some static value that you can use to get and put the item
• host - a string attribute that will be updated to reflect the name of the host that is doing the scan, each time a scan is successfully started
• lastScanTime - a numeric attribute that will be updated to the epoch timestamp when the last scan was started, each time a scan is successfully started
• version - a numeric attribute that will be used as a monotonically increasing numeric value for the purpose of optimistic concurrency (more on this below)

---

Now, on each EC2 host, set up an operation to run once every 30 seconds (can be a local cron set up to run every 30 seconds).

When the scheduled operation runs, do the following:

1. GetItem to read the current value of the scan-schedule item we just discussed above
2. If the lastScanTimestamp was more than 25 seconds ago, attempt to update the item with this host's info and set lastScanTimestamp to the current timestamp, incrementing the version attribute as well, using a conditional expression that checks that the version == the same value that was read at step 1
3. If the conditional update succeeds, the the scan operation can commence; however, if the conditional update fails, then it means another host got to it first and this host should not continue to scan

Note that the key to the algorithm above is the conditional expression which allows you to condition a read-modify-write sequence such that you can detect if somebody else happened to change the item in the time since you read the status and made your attempt to update it.