AWS ECS: 503 Service Temporarily Unavailable

Question

I have deployed a simple Express server that does SSR for an Angular 7 application. It runs without issue, but after some time, requests result in 503 Service Temporarily Unavailable responses.

The events tab in my service shows my tasks draining and restarting, but nothing else abnormal. I don't see anything strange in the instance's ecs-agent logs, other my task sending container stop/start events.

From my service's events tab:

af81a908-d0ae-4f8d-a19f-4f5723053c71
2019-03-26 13:15:28 -0700
service ui has started 1 tasks: task 61e77740-7b6a-4785-a5c8-9e7581a79745.
8bc7eb29-6a22-4af8-9994-57f2cfd7ad26
2019-03-26 13:14:55 -0700
service ui has begun draining connections on 1 tasks.
cb064206-0ed4-4b68-b528-957175a93196
2019-03-26 13:14:55 -0700
service ui deregistered 1 targets in target-group contrast-fm-target-group
398e8407-17fa-466e-a482-1840f7b9674c
2019-03-26 11:40:31 -0700
service ui has reached a steady state.

From the ecs-agent log on my instance:

2019-03-26T20:41:02Z [INFO] Managed task [arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea]: sending task change event [arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-03-26 20:40:56.5934327 +0000 UTC m=+24378.048113509, PullStoppedAt: 2019-03-26 20:40:56.769284895 +0000 UTC m=+24378.223965695, ExecutionStoppedAt: 2019-03-26 20:41:02.41672748 +0000 UTC m=+24383.871408417]
2019-03-26T20:41:02Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea contrast-ui -> STOPPED, Exit 1, , Ports [{3000 80 0.0.0.0 0}], Known Sent: RUNNING
2019-03-26T20:41:02Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-03-26 20:40:56.5934327 +0000 UTC m=+24378.048113509, PullStoppedAt: 2019-03-26 20:40:56.769284895 +0000UTC m=+24378.223965695, ExecutionStoppedAt: 2019-03-26 20:41:02.41672748 +0000 UTC m=+24383.871408417, arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea contrast-ui -> STOPPED, Exit 1, , Ports [{3000 80 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-03-26T20:41:02Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-03-26 20:40:56.5934327 +0000 UTC m=+24378.048113509, PullStoppedAt: 2019-03-26 20:40:56.769284895 +0000 UTC m=+24378.223965695, ExecutionStoppedAt: 2019-03-26 20:41:02.41672748 +0000 UTC m=+24383.871408417, arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea contrast-ui -> STOPPED, Exit 1, , Ports [{3000 80 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-03-26T20:41:02Z [INFO] Managed task [arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea]: sent task change event [arn:aws:ecs:us-west-2:584648718741:task/13cfb2bc-d40a-4c5b-be47-f73dc2ed28ea -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-03-26 20:40:56.5934327 +0000 UTC m=+24378.048113509, PullStoppedAt: 2019-03-26 20:40:56.769284895 +0000 UTC m=+24378.223965695, ExecutionStoppedAt: 2019-03-26 20:41:02.41672748 +0000 UTC m=+24383.871408417]
2019-03-26T20:41:02Z [WARN] DockerGoClient: Unable to decode stats for container d999c0b9afba34f3e7e77e5d34a29c64edb931b1fbc99f53ccfe403c4490db9f: context canceled
2019-03-26T20:41:02Z [INFO] Container d999c0b9afba34f3e7e77e5d34a29c64edb931b1fbc99f53ccfe403c4490db9f is terminal, stopping stats collection
2019-03-26T20:41:04Z [INFO] Saving state! module="statemanager"

Answer 1

Seems that, for some reason, your Application Load Balancer is not responding to the ECS health check calls.

Check the if your Target Group configuration is correctly pointing to a route that returns HTTP 200 . Also, check if your Security Group from your Service is not blocking traffic to the port that your application is running.

More on this here .