Kubernetes ImagePullBackOff with Private Registry on Docker Hub

Question

I have a private Docker Hub registry with a (rather large) image in it that I control.

I also have a Helm deployment chart that specifies an imagePullSecret , after having followed the instructions here https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ .

No matter what I do, though, when installing the Helm chart, I always end up with the following (taken from kubectl describe pod <pod-id> ):

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  26m                  default-scheduler  Successfully assigned default/<release>-69584657b7-vkps6 to <node>
  Warning  Failed     6m28s (x3 over 20m)  kubelet            Failed to pull image "<registry-username>/<image>:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/<registry-username>/<image>:latest": failed to copy: httpReadSeeker: failed open: server message: invalid_token: authorization failed
  Warning  Failed     6m28s (x3 over 20m)  kubelet            Error: ErrImagePull
  Normal   BackOff    5m50s (x5 over 20m)  kubelet            Back-off pulling image "<registry-username>/<image>:latest"
  Warning  Failed     5m50s (x5 over 20m)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    5m39s (x4 over 26m)  kubelet            Pulling image "<registry-username>/<image>:latest"

I have looked high and low on the internet for answers pertaining to this invalid_token output, but have yet to find anything concrete.

I have verified that I can run docker pull manually with the image in question both on the K8s node as well as other boxes. It works just fine.

I have tried using docker.io as the repository URI, as well as (the recommended) https://index.docker.io/v1/ .

I have tried using my own Docker Hub password as well as a generated Personal Access Token (I can actually see in Docker Hub that the PAT was, in fact, used, despite the pull failing).

I've examined the secrets via kubectl to verify they're of the expected format and contain the correct data (username, password/token, etc.). They're all fine and match what I'd get when I run docker login on the command line.

I have used this node to deploy other releases via Helm and they have all worked fine (although at least one has been from a different registry).

I am relatively new to K8s and Helm, but I've used Docker for a long while now and I'm at a loss as to this invalid_token issue.

Any help would be greatly appreciated.

Thank you in advance.

UPDATE

Here's the (sanitized) output of helm template :

---
# Source: <deployment>/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: release-name-<deployment>
  labels:
    helm.sh/chart: <deployment>-0.1.0
    app.kubernetes.io/name: <deployment>
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: <deployment>
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels:
        app.kubernetes.io/name: <deployment>
        app.kubernetes.io/instance: release-name
    spec:
      imagePullSecrets:
        - name: regcred-docker-pat
      securityContext:
        {}
      containers:
        - name: <deployment>
          securityContext:
            {}
          image: "<registry-username>/<image>:latest"
          imagePullPolicy: IfNotPresent
          resources:
            {}

Answer 1

Is the imagepullsecret created by the helm chart?
Is the imagepullsecret available when the deployment is created? Do you apply the deployment before the imagepullsecret is available?

I remember the order matters when applying the imagepullsecret; the kube-api does not retry pulling after failure because of authentication.