Kubernetes containerd failed to pull images from private registry

Question

I have a Kubernetes cluster in azure(AKS) with kubernetes version 1.22.11. I'm unable to pull images from our private registry. I have edited config.toml like below and restarted containerd service as well. I tried this with auth as well, instead of username/password still it didn't work.

 version = 2
subreaper = false
oom_score = 0
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "mcr.microsoft.com/oss/kubernetes/pause:3.5"
  [plugins."io.containerd.grpc.v1.cri".containerd]



    [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
      runtime_type = "io.containerd.runtime.v1.linux"
      runtime_engine = "/usr/bin/runc"
    [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
      runtime_type = "io.containerd.runtime.v1.linux"
      runtime_engine = "/usr/bin/runc"



  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "/opt/cni/bin"
    conf_dir = "/etc/cni/net.d"
    conf_template = "/etc/containerd/kubenet_template.conf"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."test.registry.com"]
      endpoint = ["https://test.registry.com:5000"]
  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."test.registry.com".tls]
       insecure_skip_verify=true
  [plugins."io.containerd.grpc.v1.cri".registry.configs."test.registry.com".auth]
    username = "xxxxx"
    password = "xxxxx"
[metrics]
  address = "0.0.0.0:10257"

I'm getting the below error when I try to pull an image from registry

crictl pull test.registry.com:5000/sba-housekeeping/logrotate:2.0.2 FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "test.registry.com:5000/sba-housekeeping/logrotate:2.0.2": failed to resolve reference "test.registry.com:5000/sba-housekeeping/logrotate:2.0.2": pulling from host software.openet.com:5000 failed with status code [manifests 2.0.2]: 401 Unauthorized

My credentials are correct, I have verified them through docker login

Answer 1

You can add your docker registry credentials to the cluster by creating a K8S secret of type kubernetes.io/dockerconfigjson and using it to pull the image.

To create it from a docker config file:

kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
    --type=kubernetes.io/dockerconfigjson

Or by providing the credens:

kubectl create secret docker-registry regcred \
    --docker-server=<your-registry-server> \
    --docker-username=<your-name> \
    --docker-password=<your-pword> \
    --docker-email=<your-email>

You can use it by just adding imagePullSecrets :

apiVersion: v1
kind: Pod
metadata:
  name: <pod-name>
spec:
  containers:
  - name: <container-name>
    image: <your-private-image>
  imagePullSecrets:
  - name: regcred