Kubernetes fails to pull image with unknown blob error

Question

I created a Docker image based on microsoft/dotnet-framework of a C#.NET console application built for Windows containers, then ensured I can run the image in a container locally. I successfully pushed the image to our Azure Container registry. Now I'm trying to create a deployment in our Azure Kubernetes service, but I'm getting an error:

Failed to pull image "container-registry/image:tag": rpc error: code = Unknown desc = unknown blob

I see this error on my deployment, pods, and replica sets in the Kubernetes dashboard.

We already have a secret that works with the azure-vote app, so I wouldn't think this is related to secrets, but I could be wrong.

So far, I've tried to create this deployment by pasting the following YAML into the Kubernetes dashboard Create dialog:

apiVersion:
kind: Deployment
metadata:
  name: somename
spec:
  selector:
    matchLabels:
      app: somename
      tier: backend
  replicas: 2
  template:
    metadata:
      labels:
        app: somename
        tier: backend
    spec:
      containers:
      - name: somename
        image: container-registry/image:tag
        ports:
        - containerPort: 9376

And I also tried running variations of this kubectl command:

kubectl run deploymentname --image=container-registry/image:tag

In my investigation so far, I've tried reading about different parts of k8s to understand what may be going wrong, but it's all fairly new to me. I think it may have to do with this being a Windows Server 2016 based image. A team member successfully added the azure-vote tutorial code to our AKS, so I'm wondering if there is a restriction on a single AKS service running deployments for both Windows and Linux based containers. I see by running az aks list that the AKS has an agentPoolProfile with "osType": "Linux", but I don't know if that means simply that the orchestrator is in Linux or if the containers in the pods have to be Linux based. I have found stackoverflow questions about the "unknown blob" error, and it seems the answer to this question might support my hypothesis, but I can't tell if that question is related to my questions.

Since the error has to do with failing to pull an image, I don't think this has to do with configuring a service for this deployment. Adding a service didn't change anything. I've tried rebuilding my app under the suspicion that the image was corrupted, but rebuilding and re-registering had no effect. Another thing that doesn't seem relevant that I read about is this question and answer regarding a manifest mismatch (which I don't completely understand yet).

I have not tried creating a local Kubernetes. I don't know if that's something folks typically do.

Summary of questions:

1. What causes this unknown blob error? Does it have to do with a Windows container/Linux container mismatch?
2. Does the agent pool profile affect all the nodes in the cluster, or just the "master" nodes?

Let me know if you need more information. Thanks.

Answer 1

1. What causes this unknown blob error? Does it have to do with a Windows container/Linux container mismatch? It's because you're trying to run a Windows-based Docker container on a Linux host. It has nothing directly to do with Kubernetes or AKS. Currently AKS is in preview and supports only Linux environments. To be more precise, when you provision your AKS cluster ( az aks create ), all your k8s minions (worker nodes) will be Linux boxes and thus will not be able to run Windows-based containers.

2. Does the agent pool profile affect all the nodes in the cluster, or just the "master" nodes? It affects the worker nodes and is used to group them together logically so you can better manage workload distribution. In the future, when AKS supports both Linux and Windows, you will be able to ie create agent pools based on OS type and instruct k8s to deploy your Windows-based services only to the Windows-based hosts (agents).

Answer 2

I'm not sure if this applies to you, because it may seem to be a new deployment but I had this for existing ones twice now in a GKE environment. I'm using an Ubuntu image with nodejs, all Linux based. For some reason it seems the image is damaged or something in the GKE setup thinks that. I can resolve the problem by pushing again to the registry and then point the deployment to the new image.

I can reproduce it by simply pulling the docker image from another authenticated machine like:

docker pull gcr.io/project-name/app-name@sha256:somelongrandomhash

That gives me the same error. Once I noticed that, I just repushed the image and it worked. So although it might be the OS base image difference causing it as others suggested, check a local pull to see if that works well.