ECS 任务只能从 SQS 队列中选择一条消息

Question

I have an architecture which looks like that:

• As soon as a message is sent to a SQS queue, an ECS task picks this message and process it.
• Which means that if X messages are sent into the queue, X ECS task will be spun up in parallel. An ECS task is only able to fetch one message (per my code above)

The ECS task uses a dockerized Python container, and uses boto3 SQS client to retrieve and parse the SQS message:

sqs_response = get_sqs_task_data('<sqs_queue_url>')
sqs_message = parse_sqs_message(sqs_response)

while sqs_message is not None:
    # Process it
    # Delete if from the queue
    
    # Get next message in queue
    sqs_response = get_sqs_task_data('<sqs_queue_url>')
    sqs_message = parse_sqs_message(sqs_response)

def get_sqs_task_data(queue_url):
    client = boto3.client('sqs')

    response = client.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=1
    )

    return response

def parse_sqs_message(response_sqs_message):

    if 'Messages' not in response_sqs_message:
        logging.info('No messages found in queue')
        return None
    
    # ... parse it and return a dict

    return {
        data_1 = ..., 
        data_2 = ...
    }

All in all, pretty straightforward.

In get_sqs_data() , I explicitely specify that I want to retrieve only one message (because 1 ECS task has to process only one message). In parse_sqs_message() , I test if there are some messages left in the queue with

if 'Messages' not in response_sqs_message:
        logging.info('No messages found in queue')
        return None

When there is only one message in the queue (meaning one ECS task has been triggered), everything is working fine. The ECS task is able to pick the message, process it and delete it.

However, when the queue is populated with X messages ( X > 1 ) at the same time , X ECS task are triggered, but only ECS task is able to fetch one of the message and process it.
All the others ECS tasks will exit with No messages found in queue , although there are X - 1 messages left to be processed.

Why is that? Why are the others task not able to pick the messages left to be picked?

If that matters, the VisibilityTimeout of SQS is set to 30mins.

Any help would greatly be appreciated!

Feel free to ask for more precision if you want so.

Answer 1

I forgot to give an answer to that question.

The problem was the fact the the SQS was setup as a FIFO queue. A FIFO Queue only allows one consumer at a time (to preserve the order of the message). Changing it to a normal (standard) queue fixed this issue.

Answer 2

I'm not sure to understand how the tasks are triggered from SQS, but from what I understand in the SQS SDK documentation, this might happen if the number of messages is small when using short polling. From the get_sqs_task_data definition, I see that your are using short polling.

Short poll is the default behavior where a weighted random set of machines is sampled on a ReceiveMessage call. Thus, only the messages on the sampled machines are returned. If the number of messages in the queue is small (fewer than 1,000), you most likely get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response. If this happens, repeat the request.

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.receive_message

You might want to try to use Long polling with a value superior to the visibility timeout

I hope it helps