如何让 Docker Swarm 管理器使用 IAM 角色权限从 AWS ECR 中提取图像？

Question

I'm having trouble pulling images from AWS ECR, running Docker Swarm. It's been working ok for years, but my swarm manager nodes were changed to new EC2 instances. Now my services fail to deploy:

~ $ docker stack deploy -c dkr_compose_geo_site:3.2.0 --with-registry-auth geo_stack

The manager node log shows "no basic auth credentials":

May 19 21:21:12 ip-172-31-3-108 root: time="2020-05-19T21:21:12.857007050Z" level=error msg="pulling image failed" error="Get https://445523.dkr.ecr.us-west-2.amazonaws.com/v2/geo_site/manifests/sha256:da5820742cd0ecd52e3a2c61179a039ce80996564604b70465e3966087380a09: no basic auth credentials" module=node/agent/taskmanager node.id=eix8c6orbunemismg03ib1rih service.id=smilb788pets7y5rgbu3aze9l task.id=zd3ozdpr9exphwlz318pa9lpe 
May 19 21:21:12 ip-172-31-3-108 root: time="2020-05-19T21:21:12.857701347Z" level=error msg="fatal task error" error="No such image: 445523.dkr.ecr.us-west-2.amazonaws.com/geo_site@sha256:da5820742cd0ecd52e3a2c61179a039ce80996564604b70465e3966087380a09" module=node/agent/taskmanager node.id=eix8c6orbunemismg03ib1rih service.id=smilb788pets7y5rgbu3aze9l task.id=zd3ozdpr9exphwlz318pa9lpe 

This manager node is running on an EC2 Instance with an IAM Role; the IAM Role has an ECR policy that appears to grant permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

From reading the AWS/Docker docs, I thought docker commands run on a manager node should adopt the Instance IAM Role and access the ECR repo using the associated policy permissions. It's always seemed to work that way, but now it's looking like there might have been some config file hidden on the old manager node; I'm on a new instance and it doesn't work. I don't run an AWS-CLI on these manager nodes, so there's no aws ecr get-login to login manually. How do I get this new manager node to authenticate with ECR?

Thanks!

Answer 1

My solution, based on comment by Luigi Lopez and amazon-ecr-credential-helper :

The AWS IAM Role allows authentication, but the docker cli must still present credentials to the ECR, as Luigi pointed out in his comment.

This is a Docker Swarm implementation, with nodes running the Alpine OS. There is an aws-cli package available for Alpine, but the installation took a lot of fussing around and in the end the binary crashed anyway.

The Amazon ECR Credential Helper is a better long-term solution in any case because you don't need to get new tokens every 12 hours or set up a proxy server, etc. It uses the recommended IAM Role authentication, with no credentials stored on the machine or leaking into log files.

So under Alpine I followed the instructions in the link above to build from sources. I installed go, git, and make, and then built the credential-helper as described. I set up the PATH as described, created a config file, and then my deployment worked. There's no docker login required.