How to set Kubernetes image pull retry limit

Question

Kubernetes ends up with long running pods when an image specified for a container is purged from an image repository. These deployments are created by a continuous integration system and sometimes pipelines are run or rerun when images have been purged.

The status from kubectl get pods shows ImagePullBackOff .

What should be set in the kube config yaml file to stop these pods from running for days? Ideally we just want the Image to be pulled a couple of times and then fail if it's unsuccessful.

The pod definition is

apiVersion: v1
kind: Pod
metadata:
  name: test-missing-image

spec:
  containers:

  - image: missingimage

    name: test
    resources:
      limits:
        memory: "10000Mi"
    readinessProbe:
      httpGet:
        port: 5678
        path: /somePath
      initialDelaySeconds: 360
      periodSeconds: 30
      timeoutSeconds: 30

  restartPolicy: Never
  terminationGracePeriodSeconds: 0

Thanks!

Answer 1

AKAIK, the only way to control this as of this writing is with the imagePullPolicy in the container spec.

You may set it to Never but your pod will not run since the image is not present locally. Or you can set it to IfNotPresent but somehow you will have to have to create an image with that specific tag locally in your K8s nodes. Either option is not ideal, but I believe there might be a rationale to have it go into ImagePullBackOff : people would want to to know why their pod is not running.

So IMO the bigger question is why would you want to delete/invalidate images in your docker registry that are still running in your cluster? Why not update the pods/deployments/daemonsets/replicasets/statefulsets with the latest images prior to deleting or invalidating an image in the docker registry (also called deploy)?

The general practice could be something like this:

create new image => deploy it => make sure everything is ok => 
{
  ok => invalidate the old image tag.
  not ok => rollback => delete new image tag => go back to create new image => create new image tag.
}

Note, layers, and images are not deleted in a docker registry. You can delete or overwrite tags: How to delete images from a private docker registry?