AWS ECS Task single instance

Question

In my architecture when I receive a new file on S3 bucket, a lambda function triggers an ECS task. The problem occurs when I receive multiple files at the same time: the lambda will trigger multiple instance of the same ECS task that acts on the same shared resources.

I want to ensure only 1 instance is running for specific ECS Task, how can I do? Is there a specific setting that can ensure it?

I tried to query ECS Cluster before run a new instance of the ECS task, but (using AWS Python SDK) I didn't receive any information when the task is in PROVISIONING status, the sdk only return data when the task is in PENDING or RUNNING.

Thank you

Answer 1

I don't think you can control that because your S3 event will trigger new tasks. It will be more difficult to check if the task is already running and you might miss execution if you receive a lot of files.

You should think different to achieve what you want. If you want only one task processing that forget about triggering the ECS task from the S3 event. It might work better if you implement queues. Your S3 event should add the information (via Lambda, maybe?) to an SQS queue.

From there you can have an ECS service doing a SQS long polling and processing one message at a time.