Target accessible but ECS service logs says Request timed out

Question

I am just getting started with ECS. I created a docker image which contains a simple web-application that runs on port 9000. Uploaded this to ECR and started deployment following the documentation. I am using Fargate.

I defined the task, created a cluster > service with appropriate task version. I have specified this in health check

"CMD-SHELL","curl -f http://localhost:9000/health/check || exit 1"

I have checked the above end-point locally and it works perfectly fine.

I am using ALB and from there I have checked that the target group is configured correctly and I even checked the targets via my bastion host and the health end-point is working fine on the target.

curl -f http://<target-ip>:9000/health/check

But when I see the service events, I see the following entry very frequently. And I cannot access the web app using ALB url.

02434c5e-00f7-4e9a-8716-480791cb164a
2022-12-08 13:15:10 +0530
service hello-service (port 9000) is unhealthy in target-group ecs-pankaj-hello-service-1 due to (reason Request timed out).
1048420a-7e07-42d6-a76b-487989ea280e
2022-12-08 13:12:26 +0530
service hello-service registered 1 targets in target-group ecs-pankaj-hello-service-1
4991efc7-d54c-4984-a2dc-da71a6dc8200
2022-12-08 13:11:08 +0530
service hello-service has started 1 tasks: task b89d8cbf4bad48f2871f5984b4dc2825.
a7002253-53ef-4212-bfcb-f195c4062a42
2022-12-08 13:10:59 +0530
service hello-service deregistered 1 targets in target-group ecs-pankaj-hello-service-1
6c69bda9-792d-434f-9748-878262206e45
2022-12-08 13:10:59 +0530
(service hello-service, taskSet ecs-svc/3722148168959484462) has begun draining connections on 1 tasks.
c1b94455-057a-4b3c-bf5a-df1b3f247474
2022-12-08 13:10:59 +0530
service hello-service deregistered 1 targets in target-group ecs-pankaj-hello-service-1
ffc4387b-1de5-46ee-8a3e-efde12c16e4a
2022-12-08 13:10:49 +0530
service hello-service has stopped 1 running tasks: task ead2367371cf44c2851e1c44bb46ffd0.

Result: I cannot access the web app via ALB url using a browser.

Answer 1

If you receive either of the following errors:

(service AWS-service) (port 8080) is unhealthy in (target-grouparn:uxyztargetgroup/aws-targetgroup/123456789) due to (reason Health checks failed with these codes: [502]) or [request timeout] (service AWS-Service) (port 8080) is unhealthy in target-group tf-20190411170 due to (reason Health checks failed) Try these troubleshooting steps:

If your container is mapped to port 80, confirm that your container security group allows inbound traffic on port 80 for the load balancer. Confirm that the ping port value for your load balancer health is configured correctly. If this port isn't configured correctly, then your load balancer could de-register the container from itself. Define a minimum health check grace period. This instructs the service scheduler to ignore Elastic Load Balancing health checks for a pre-defined time period after a task has been instantiated. Monitor the CPU and memory metrics of the service. For example, high CPU can make your application unresponsive and result in a 502 error. Check your application logs for application errors. Check if the ping port and the health check path are configured correctly. Make sure that your backend database is connected successfully. This assumes that your application is running as a set of tasks launched by Amazon Elastic Container Service (Amazon ECS) on Amazon Elastic Compute Cloud (Amazon EC2) instances. It also assumes that your application can't communicate with the Amazon Relational Database Service (Amazon RDS) database.