Kubernetes deployment - specify multiple options for image pull as a fallback?

Question

We have had image pull issues at one time or another with all of our possible docker registries including Artifactory, AWS ECR, and GitLab. Even DockerHub occasionally has issues.

Is there a way in a Kubernetes deployment to specify that a pod can get an image from multiple different repositories so it can fall back if one is down?

If not, what other solutions are there to maintain stability? I've seen things like Harbor and Trow, but it seems like a heavy handed solution to a simple problem.

Answer 1

Is there a way in a Kubernetes deployment to specify that a pod can get an image from multiple different repositories so it can fall back if one is down?

Not really, not natively. You could probably trick a K8s node to pull images from different image registries (one at a time) if you place them behind something like a TCP load balancer that directs traffic to multiple registries. But this might take a lot of testing and work.

If not, what other solutions are there to maintain stability? I've seen things like Harbor and Trow, but it seems like a heavy handed solution to a simple problem.

I'd say either Harbor , Quay , and Trow is the way to go if you want something more redundant.

Kubernetes has the ability to set ImagePullPolicy and you can set it for example to Never if you'd like to pre-pull all your critical images on all the K8s nodes. You can tie this to some automation to pre-pull your images across your clusters and nodes.

I've actually opened a K8s feature request to see if this idea gains traction.

Update:

If you're using containerd or cri-o (or even Docker has registry mirrors). You have the ability to configure mirror registries:

containerd.toml example

...
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://localhost:32000"]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = ["https://gcr.io"]
      [plugins.cri.registry.configs]
        [plugins.cri.registry.configs.auths]
          [plugins.cri.registry.configs.auths."https://gcr.io"]
            auth = "xxxxx...."
...

cri-o.conf example

...
# registries is used to specify a comma separated list of registries to be used
# when pulling an unqualified image (e.g. fedora:rawhide).
registries = [
“registry.example.xyz”,
“registry.fedoraproject.org”
]
...

✌️