Laravel elements in queue are processed multiple times

Question

We use the Laravel Redis Queue and a supervisor setup with multiple parallel running workers. Sometimes it happens that one element in the queue get processed multiple times.

Is there a trick, like a flag i can set or something else to avoid this behavior?

Answer 1

If there is an exception or the job inexplicably fails, the job will automatically be retried. This will occur even if most of the job has already ran.

Answer 2

This is because the job messages become available for other consumers.

In your config/queue.php configuration file, each queue connection defines a retry_after option. This option specifies how many seconds the queue connection should wait before retrying a job that is being processed. For example, if the value of retry_after is set to 90, the job will be released back onto the queue if it has been processing for 90 seconds without being deleted. Typically, you should set the retry_after value to the maximum number of seconds your jobs should reasonably take to complete processing.

See https://laravel.com/docs/5.7/queues#queue-workers-and-deployment

When a consumer receives and processes a message from a queue, the message remains in the queue. The queue doesn't automatically delete the message. Because a queue is a distributed system, there's no guarantee that the consumer actually receives the message (for example, due to a connectivity issue, or due to an issue in the consumer application). Therefore, the consumer must delete the message from the queue after receiving and processing it. Immediately after a message is received, it remains in the queue. To prevent other consumers from processing the message again, you should set the retry_after value to the maximum number of seconds your jobs should reasonably take to complete processing.

For Amazon SQS:

The only queue connection which does not contain a retry_after value is SQS. SQS will retry the job based on the Default Visibility Timeout which is managed within the AWS console. Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. The default visibility timeout for a message is 30 seconds.This means that other consumers can see and pickup the message again after 30 seconds.