ErrImagePull: x509: certificate signed by unknown authority

Question

When I try to create a pod in kubernetes with my image in my Harbor registry,I got an ErrImagePull Error, which looks like that:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  10s   default-scheduler  Successfully assigned test/test-pod to ubuntu-s-2vcpu-2gb-ams3-01-slave01
  Normal   Pulling    9s    kubelet            Pulling image "my.harbor.com/test/nginx:1.18.0"
  Warning  Failed     9s    kubelet            Failed to pull image "my.harbor.com/test/nginx:1.18.0": rpc error: code = Unknown desc = failed to pull and unpack image "my.harbor.com/test/nginx:1.18.0": failed to resolve reference "my.harbor.com/test/nginx:1.18.0": failed to do request: Head https://my.harbor.com/v2/test/nginx/manifests/1.18.0: x509: certificate signed by unknown authority
  Warning  Failed     9s    kubelet            Error: ErrImagePull
  Normal   BackOff    8s    kubelet            Back-off pulling image "my.harbor.com/test/nginx:1.18.0"
  Warning  Failed     8s    kubelet            Error: ImagePullBackOff

I think the crucial problem is that 'x509: certificate signed by unknown authority but I really don't know what's wrong, since I copied my CA to both kubernetes master node and slave node , and they can both login to harbor and run docker pull my.harbor.com/test/nginx:1.18.0 to pull the image successfully.

I had been bothered days for this, any reply would be grateful.

Answer 1

I copied the ca.crt to /etc/docker/certs.d/my.harbor.com/

This will make it work for the docker engine, which you've shown.

along with my.harbor.cert and my.harbor.com.key

I'd consider that a security violation and no longer trust the secret key for your harbor host. The private key should never need to be copied off of the host.

and I also copied the ca.crt to /usr/local/share/ca-certificates/ and run command update-ca-certificates to update.

That's the step that should have resolved this.

You can verify that you loaded the certificate with:

openssl s_client -connect my.harbor.com:443 -showcerts </dev/null

If the output for that doesn't include a message like Verification: OK , then you didn't configure the host certificates correctly and need to double check the steps for your Linux distribution. It's important to check this on each of your nodes. If you only update the manager and pull your images from a worker, that worker will still encounter TLS errors.

If openssl shows a successful verification, then check your Kubernetes node. Depending on the CRI, it could be caching old certificate data and need to be restarted to detect the change on the host.

As for CRI, I don't know what is it

Container Runtime Interface, part of your Kubernetes install. By default, this is containerd on many Kubernetes distributions. containerd and other CRI's (except for docker-shim ) will not look at the docker configuration.