401 Unauthorized error while trying to pull image from Google Container Registry

Question

I am using google container registry (GCR) to push and pull docker images. I have created a deployment in kube.netes with 3 replicas. The deployment will use a docker image pulled from the GCR.

Out of 3 replicas, 2 are pulling the images and running fine.But the third replica is showing the below error and the pod's status remains "ImagePullBackOff" or "ErrImagePull"

"Failed to pull image "gcr.io/xxx:yyy": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/xxx:yyy": failed to resolve reference "gcr.io/xxx:yyy": unexpected status code: 401 Unauthorized"

I am confused like why only one of the replicas is showing the error and the other 2 are running without any issue. Can anyone please clarify this?

Thanks in Advance!

Answer 1

ImagePullBackOff and ErrImagePull indicate that the image used by a container cannot be loaded from the image registry.

401 unauthorized error might occur when you pull an image from a private Container Registry repository. For troubleshooting the error:

1. Identify the node that runs the pod by kubectl describe pod POD_NAME | grep "Node:"kubectl describe pod POD_NAME | grep "Node:"

2. Verify the node has the storage scope by running the command

    gcloud compute instances describe NODE_NAME --zone=COMPUTE_ZONE --format="flattened(serviceAccounts[].scopes)"

3. The node's access scope should contain at least one of the following:

   serviceAccounts[0].scopes[0]: https://www.googleapis.com/auth/devstorage.read_only serviceAccounts[0].scopes[0]: https://www.googleapis.com/auth/cloud-platform

4. Recreate the node pool that node belongs to with sufficient scope and you cannot modify existing nodes, you must recreate the node with the correct scope.

   • Create a new node pool with the gke-default scope by the following command

     gcloud container node-pools create NODE_POOL_NAME --cluster=CLUSTER_NAME --zone=COMPUTE_ZONE --scopes="gke-default"

   • Create a new node pool with only storage scope

     gcloud container node-pools create NODE_POOL_NAME --cluster=CLUSTER_NAME --zone=COMPUTE_ZONE --scopes="https://www.googleapis.com/auth/devstorage.read_only"

Refer to the link for more information on the troubleshooting process.

Answer 2

Hi you will setup role for cluster to access GCR images for pulling and pushing you can see https://github.com/GoogleContainerTools/skaffold/issues/336