kubernetes fails to pull kube-proxy from gcr.io

Question

Suddenly today my Kubernetes cluster seems to be unable to pull the kube-proxy image from gcr.io;

Failed to pull image "gcr.io/google_containers/kube-proxy:2691d6fbc11c57b4d1d4cd831125eb4b": image pull failed for gcr.io/google_containers/kube-proxy:2691d6fbc11c57b4d1d4cd831125eb4b, this may be because there are no credentials on this request. details: (Tag 2691d6fbc11c57b4d1d4cd831125eb4b not found in repository gcr.io/google_containers/kube-proxy)

I'm not sure that I believe the 'no credentials' part as this has been working for many months.

Anyone seen this?

Answer 1

You can try this trouble shooting skill. Log in to your k8s minion node via ssh. Type 'docker pull gcr.io/google_containers/kube-proxy:2691d6fbc11c57b4d1d4cd831125eb4b' and see what's the result.

You also have to make sure tag '2691d6fbc11c57b4d1d4cd831125eb4b' exist in gcr.io. I think this tag is too long ?

Answer 2

As suggested, you can see the problem if you try and docker pull that image yourself:

$ docker pull gcr.io/google_containers/kube-proxy:2691d6fbc11c57b4d1d4cd831125eb4b
Pulling repository gcr.io/google_containers/kube-proxy
Tag 2691d6fbc11c57b4d1d4cd831125eb4b not found in repository gcr.io/google_containers/kube-proxy

An image with the tag you specific ( 2691d6fbc11c57b4d1d4cd831125eb4b ) doesn't exist on gcr.io.

You need to use an existing tag, you can get a list of tags from the API:

curl https://gcr.io/v2/google_containers/hyperkube/tags/list 

This provides us with (eg) v1.6.0-alpha.0 :

$ docker pull gcr.io/google_containers/kube-proxy:v1.6.0-alpha.0
v1.6.0-alpha.0: Pulling from google_containers/kube-proxy
..
..