How accurate is the ApproximateReceiveCount from SQS

Question

I have a system where multiple workers consume from a standard SQS queue in parallel.

I noticed that when I have a relatively large number of messages (ie 3 millions), my total processed count at the end always result in a few more messages (around 30) than the total number of messages. (0.001% ~ 0.002% more)

I suspect it is because of the "At-least-once" delivery:

Amazon Doc: It is possible you will receive a message even after you have deleted it. This might happen on rare occasions if one of the servers storing a copy of the message is unavailable when you request to delete the message. The copy remains on the server and might be returned to you again on a subsequent receive request. You should create your system to be idempotent so that receiving a particular message more than once is not a problem.

Therefore, I'd like to use "ApproximateReceiveCount" to determine if my message has been processed before processing it:

(Worker pseudocode)

List messages = sqs.receiveMessage()
for m in messages:
   if m.approximateReceiveCount > 1 then
      skip process
   else
      process as usual
end

I'm wondering how accurate is this "approximateReceiveCount" and whether it is a good idea to have my de-duplication logic depend on it.

Note that:

I have setup a Dead Letter queue to handle any message that took longer than "Default Visibility Timeout" (set to 1h). Since no message was put back to the Dead Letter, I assume that the extra counts are not due to this "timeout" effet.

Answer 1

You cannot reliably use the approximateReceiveCount property to de-duplicate messages. Because if you receive a message, then fail, your approximateReceiveCount could be 1, but the message still needs to be processed again.

When working with SQS, best practice is to ensure your SQS message processing is idempotent . Meaning that processing the same message multiple times will yield the same results.

What this means exactly depends on your business logic.

1. You could track the SQS message IDs to determine if they have been processed already.
2. Or you could have some other ID within the message that determines whether the message has been processed already.
3. Or you could process the data multiple times, each time reaching the same result.

Solution 1 or 2 could be hard to reliably implement due to possible race conditions between processing, tracking, and possible failure.

Solution 3 may be best because you may not be able to do 1 or 2 transactually in case of a processing failure.

Issues with solutions 1 or 2

Example 1:

Suppose your logic is as follows:

1. Receive message from queue
2. Process message
3. Record message for de-duplication
4. Delete message from queue

However, if you fail between steps 2 and 3, or the message is received a second time by another processor between steps 2 and 3, then your de-duplication logic has failed.

Example 2:

Alternatively, suppose your logic is as follows:

1. Receive message from queue
2. Record message for de-duplication
3. Process message
4. Delete message from queue

Now, if you fail after step 2, or during step 3 (meaning processing never completes correctly), then you will never get another proper processing of your message.