Amazon ECS with GitHub Actions Stuck on Deploying ECS Task Definition

Question

I am using the Amazon ECS "Deploy Task Definition" Action for GitHub Actions template to deploy my app. It sometimes get stuck on this step:

Deploy Amazon ECS task definition

It can get stuck for a long time, between 30 - 60 minutes , before terminating. I am able to login to the AWS console in ECS and see the Task Definition it has created, and then Run a new Task from the newly created Task Definition manually. The result is good as I see the task running succesfully.

When I look at the logs as to why it got stuck in the first place, I have seen error messages such as:

(service myService) was unable to place a task because no container instance met all of its requirements. The closest matching (container-instance id23244214) has insufficient CPU units available.

(service myService) was unable to place a task because no container instance met all of its requirements. The closest matching (container-instance id23244214) has insufficient memory available.

I have 2 vCPUs and 2 (GiB) of memory on my EC2 instance and my task definition does not go over that limit. Since I can just create the task in the ECS dashboard, it doesn't seem like the EC2 instance is lacking memory or VCPU.

Finally, when I see an error message like the two above, I try to redeploy again via GitHub. The GitHub Action passes all its jobs as normal, but it again gets stuck on the Task Definition Step. I then stop receiving any event logs after the last memory/cpu insufficient allocation in Amazon ECS console.

It eventually fails with the error message:

Error: Resource is not in the state servicesStable

Here is my Task

{
  "ipcMode": null,
  "executionRoleArn": null,
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": null,
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 3000
        }
      ],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": null,
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "container"
    }
  ],
  "placementConstraints": [],
  "memory": "1957",
  "taskRoleArn": null,
  "family": "family",
  "pidMode": null,
  "requiresCompatibilities": [
    "EC2"
  ],
  "networkMode": null,
  "runtimePlatform": null,
  "cpu": "2048",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

What could be causing this?

Answer 1

I had a similar error (Error: Resource is not in the state servicesStable) that was solved after I increased the memory on my task definition. My image size was ~600mb, it didn't work until I changed my task definition memory to 2GB.