ECS service cannot pull from ECR

Question

ECS Fargate(ver 1.4.0) service creation is getting failed with below error

Resource initilization error unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth
Request error caused by post https://api.ecr.us-XX.amazon.aws.com timeout

Things i checked

1. ECS security group has all traffic allowed
2. VPC endpoint is created properly for ecr.api and ecr.dkr
3. ECS task execution role has permission to fetch image form ECR

Answer 1

I had this very same issue, (thought) I had the very same checks covered and this error was driving me crazy.

I actually solved it by realizing my security group for ECS tasks was limited to my VPC's CIDR block, even though the tasks will in a private subnet it needs full open egress and ingress access so it can access ECR, since it goes through the outworld internet.

So instead of something like:

ingress {
 protocol         = "tcp"
 from_port        = var.container_port
 to_port          = var.container_port
 cidr_blocks      = [var.vpc_cidr_block] #or cap later from ALB and ILB
}

You should have you SG for tasks opened like:

ingress {
  protocol         = "tcp"
  from_port        = var.container_port
  to_port          = var.container_port
  cidr_blocks      = ["0.0.0.0/0"] 
}

And that's for both egress and ingress. Matter of fact after I added this I removed the VPC Endpoints and it still worked, they're not even needed. The role for grabing images is still necessary, though I just used AWS managed AmazonECSTaskExecutionRolePolicy .

Answer 2

Found the problem! https://aws.amazon.com/blogs/compute/setting-up-aws-privatelink-for-amazon-ecs-and-amazon-ecr/ This blog explains how to configure, and I forgot to configure another one thing: Private DNS. I had to enable it on ECR (DKR and API), and CloudWatch so it could reach put the service via FQDN.