AWS ECS Fargate task parallelization with SQS

Question

I have an SQS FIFO Queue in which I'm sending messages every time I want to process data. I sent batches of data 1-2 times a day and I want to process them with ECS Fargate all together because every message I process takes 10 minutes. Also every time I consume a message the ECS task closes.

So my plan is to find a way to trigger new Tasks based on SQS Available messages.

I found this article for ECS Fargate parallelization with SQS

1. https://medium.com/nbtl/deploy-low-cost-ecs-tasks-based-on-sqs-queue-size-with-aws-cdk-8f5a47fc529d
2. https://peritossolutions.com/aws/scaling-aws-ecs-fargate-on-application-events-with-sqs-lambda/

both refer to ECS Services that I'm not using due to the Desired Count variable which will make always a task run and this will lead to an unnecessary cost.

Any suggestions on how I can implement that?

Answer 1

both refer to ECS Services that I'm not using due to the Desired Count variable which will make always a task run and this will lead to an unnecessary cost.

ECS Services is exactly what you need to use here. Desired count can be 0 when there are no messages in the queue, which will prevent any charges from being incurred. You should create an ECS service that polls the SQS queue, and configure the ECS auto-scaling to scale from 0 to N service instances based on the available message count in the queue.