为什么在Kubernetes Registry插件中无法在Pod中提取私有映像？

Question

I am very new to Kubernetes and I setup Kubernetes Registry addons just copy and pasting the yaml from Kubernetes Registry Addon just a small change in ReplicationController with emptyDir

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry-upstream
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry-upstream
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry-upstream
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
       - name: image-store
         emptyDir: {}

Then I forward the 5000 port as follows

$POD=$(kubectl get pods --namespace kube-system -l k8s-app=kube-registry-upstream \
            -o template --template '{{range .items}}{{.metadata.name}} {{.status.phase}}{{"\n"}}{{end}}' \
            | grep Running | head -1 | cut -f1 -d' ')


$kubectl port-forward --namespace kube-system $POD 5000:5000 &

I can push my images fine as follows

$docker tag alpine localhost:5000/nurrony/alpine
$docker push localhost:5000/nurrony/alpine

Then I write a Pod to test it like below

Version: v1
kind: Pod
metadata:
  name: registry-demo
  labels:
    purpose: registry-demo
spec:
  containers:
    - name: registry-demo-container
      image: localhost:5000/nurrony/alpine
      command: ["printenv"]
      args: ["HOSTNAME", "KUBERNETES_PORT"]
      env:
      - name: MESSAGE
        value: "hello world"
      command: ["/bin/echo"]
      args: ["$(MESSAGE)"]

It is throwing an error

Failed to pull image "localhost:5000/nurrony/alpine": image pull failed for localhost:5000/nurrony/alpine:latest, this may be because there are no credentials on this request. details: (net/http: request canceled)

Any idea why is this happening? Thanks in advance.

Answer 1

Most likely your proxy is not working.

The Docker Registry K8S addon comes with DaemonSet which defines registry proxy for every node which runes your kubelets. What I would suggest you is to inspect those proxies since they will map Docker Registry (K8S) Service to localhost:5000 on every node.

Please note, that even if you have green check mark on your registry proxies that does not mean they work correctly. Open the logs of them and make sure that everything is working.

If your proxy is configured and you are still getting this error then most likely environment variable REGISTRY_HOST inside kube-registry-proxy is wrong. Are you using DNS here like in example? Is your DNS configured correctely? Is it working if you put this variable to ClusterIP of your service?

Also, please be aware that your RC labels need to match SVC selectors, otherwise service cannot discover your pods.

Hope it helps.