x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error") in Kubernetes

Question

I have already deployed a local registry which listens on 192.168.xx.xx:5000 .

In /etc/hosts I have added:

192.168.xx.xx my.local.registry

and using sudo vim /etc/docker/daemon.json I have added:

{ "insecure-registries":["my.local.registry:5000"] }

Then I pushed an image on it using:

sudo docker tag hello-world my.local.registry:5000/hello-world
sudo docker push my.local.registry:5000/hello-world

Everything works as excpected. In https://my.local.registry:5000/v2/_catalog I am able to see the pushed image:

{"repositories":["hello-world"]}

In the next step, I wanted to create a pod, thus a Deployment which will be able to download the image from my local registry. Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry-test
  labels:
    app: registry-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry-test
  template:
    metadata:
      labels:
        app: registry-test
    spec:
      containers:
      - name: registry-test
        image: my.local.registry:5000/hello-world

I have generated my own certificate using:

openssl req -newkey rsa:4096 -nodes -sha256 -keyout ./certs/tls.key -x509 -days 365  -subj "/C=GR/ST=./L=./O=./CN=my.local.registry" -addext "subjectAltName = DNS:my.local.registry" -out ./certs/tls.crt

and then I created a folder sudo mkdir -p /etc/docker/certs.d/my.local.registry:5000 where I put the newly created certificate using sudo scp certs/tls.crt /etc/docker/certs.d/my.local.registry:5000/ca.crt

Then I added sudo cp certs/tls.crt /usr/local/share/ca-certificates/ca.crt and finally I executed:

sudo update-ca-certificates 
sudo service docker restart
sudo systemctl restart containerd

However, when I apply the Deployment with kubectl apply -f mytestDeployment.yaml I get

Failed to pull image "my.local.registry:5000:5000/hello-world": rpc error: code = Unknown desc = failed to pull and unpack image "my.local.registry:5000:5000/hello-world:latest": failed to resolve reference "my.local.registry:5000:5000/hello-world:latest": failed to do request: Head "https://my.local.registry:5000:5000/v2/hello-world/manifests/latest": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "my.local.registry:5000")

There are plenty of answers in SO regarding this matter, however I am not able to fix this. Does anyone know what am I missing here?

UPDATE

I am also using a DeamonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: registry-ca
  namespace: ches
  labels:
    k8s-app: registry-ca
spec:
  selector:
    matchLabels:
      name: registry-ca
  template:
    metadata:
      labels:
        name: registry-ca
    spec:
      containers:
      - name: registry-ca-docker
        image: busybox
        command: [ 'sh' ]
        args: [ '-c', 'mkdir /etc/docker/certs.d/my.local.registry:5000 && cp /home/core/tls.crt /etc/docker/certs.d/my.local.registry:5000/ca.crt && exec tail -f /dev/null' ]
        volumeMounts:
        - name: etc-docker
          mountPath: /etc/docker/certs.d
        - name: ca-cert
          mountPath: /home/core
      - name: registry-ca-containerd
        image: busybox
        command: [ 'sh' ]
        args: [ '-c', 'cat /home/core/tls.crt > /home/core-containerd/ca.crt && exec tail -f /dev/null']
        volumeMounts:
        - name: ca-cert
          mountPath: /home/core
        - name: etc-containerd
          mountPath: /home/core-containerd
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etc-docker
        hostPath:
          path: /etc/docker/certs.d
      - name: ca-cert
        secret:
          secretName: ches-registry-secret
      - name: etc-containerd
        hostPath:
          path: /usr/local/share/ca-certificates

However the error persists.

Answer 1

Actually you did everything right setting up your private registry. However Kube.netes doesn't allow pulling images from insecure private registries (yes, a self-signed certificate is still considered "insecure").

I'm afraid you have to tell every Kube.netes node that your my.local.registry is either an insecure registry or place the certificate file in every node (as described in the second link of Priyanka's answer).

A different option might be to create a certificate using letsencrypt, so it's signed by a known certificate authority, thus being "secure". But that limits the use of custom domain names.