microk8s拉图像，卡在ContainerCreating状态

Question

As described in the doc (based on ).

I am on Ubuntu as a host OS.

docker --version

Docker version 18.09.6, build 481bc77

microk8s 1.14/beta

---

Enable local registry for microk2s :

microk8s.enable registry

Checking:

watch microk8s.kubectl get all --all-namespaces

container-registry pod/registry-577986746b-v8xqc 1/1 Running 0 36m

Then:

Edit:

sudo vim /etc/docker/daemon.json

Add this content:

{
  "insecure-registries" : ["127.0.0.1:32000"]
}

Restart:

sudo systemctl restart docker

Double checking, see if insecure is applied:

docker info | grep -A 2 Insecure

 Insecure Registries: 127.0.0.1:32000 127.0.0.0/8 WARNING: No swap limit support 

Tag:

docker tag my-registry/my-services/my-service:0.0.1-SNAPSHOT 127.0.0.1:32000/my-service

Checking:

docker images

127.0.0.1:32000/my-service latest e68f8a7e4675 19 hours ago 540MB

Pushing:

docker push 127.0.01:32000/my-service

I see my image here: http://127.0.0.1:32000/v2/_catalog

In deployment-local.yml I have, now the proper image:

...spec:
  containers:
    - name: my-service-backend
      image: 127.0.0.1:32000/my-service
      imagePullPolicy: Always ...

Then applying:

envsubst < ./.local/deployment-local.yml | microk8s.kubectl apply -f -

I see: ContainerCreating

By: microk8s.kubectl describe pods my-service-deployment-f85d5dcd5-cmd5

In the Events section:

Events:
  Type     Reason     Age                 From                     Message
  ----     ------     ----                ----                     -------
  Normal   Scheduled  107s                default-scheduler        Successfully assigned default/my-service-deployment-f85d5dcd5-z75tr to my-desktop
  Normal   Pulling    25s (x4 over 106s)  kubelet, my-desktop  Pulling image "127.0.0.1:32000/my-service"
  Warning  Failed     25s (x4 over 106s)  kubelet, my-desktop  Failed to pull image "127.0.0.1:32000/my-service": rpc error: code = Unknown desc = failed to resolve image "127.0.0.1:32000/my-service:latest": no available registry endpoint: failed to do request: Head https://127.0.0.1:32000/v2/my-service/manifests/latest: http: server gave HTTP response to HTTPS client
  Warning  Failed     25s (x4 over 106s)  kubelet, my-desktop  Error: ErrImagePull
  Normal   BackOff    10s (x5 over 105s)  kubelet, my-desktop  Back-off pulling image "127.0.0.1:32000/my-service"
  Warning  Failed     10s (x5 over 105s)  kubelet, my-desktop  Error: ImagePullBackOff

Seems like my-service is stuck there.

Q: What could be the reason?

UPDATE: changing all to localhost helped, meaning I could see now in the browser: http://localhost:32000/v2/_catalog .

{"repositories":["my-service"]}

But it worked only in Firefox.. weird. In chrome is pending..

Still:

Events:
  Type     Reason          Age                    From                    Message
  ----     ------          ----                   ----                    -------
  Normal   Scheduled       161m                   default-scheduler       Successfully assigned default/my-service-deployment-6d4c5587df-72xvd to my-laptop
  Normal   Pulling         160m (x4 over 161m)    kubelet, my-laptop  Pulling image "localhost/my-service"
  Warning  Failed          160m (x4 over 161m)    kubelet, my-laptop  Failed to pull image "localhost/my-service": rpc error: code = Unknown desc = failed to resolve image "localhost/my-service:latest": no available registry endpoint: failed to do request: Head https://localhost/v2/my-service/manifests/latest: dial tcp 127.0.0.1:443: connect: connection refused
  Warning  Failed          160m (x4 over 161m)    kubelet, my-laptop  Error: ErrImagePull
  Warning  Failed          159m (x6 over 161m)    kubelet, my-laptop  Error: ImagePullBackOff
  Normal   BackOff         131m (x132 over 161m)  kubelet, my-laptop  Back-off pulling image "localhost/my-service"
  Normal   SandboxChanged  22m                    kubelet, my-laptop  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling         21m (x4 over 22m)      kubelet, my-laptop  Pulling image "localhost/my-service"
  Warning  Failed          21m (x4 over 22m)      kubelet, my-laptop  Failed to pull image "localhost/my-service": rpc error: code = Unknown desc = failed to resolve image "localhost/my-service:latest": no available registry endpoint: failed to do request: Head https://localhost/v2/my-service/manifests/latest: dial tcp 127.0.0.1:443: connect: connection refused
  Warning  Failed          21m (x4 over 22m)      kubelet, my-laptop  Error: ErrImagePull
  Warning  Failed          20m (x6 over 22m)      kubelet, my-laptop  Error: ImagePullBackOff
  Normal   BackOff         2m22s (x87 over 22m)   kubelet, my-laptop  Back-off pulling image "localhost/my-service"

Seems it tries to connect by https..

--

Ok.. had to add the port for the yml image: part. will accept the answer below. Thanks.

Answer 1

In microk8s is a big different between localhost and 127.0.0.1 . As you mentioned, you were based on this Stack case and there user were using localhost .

The main issue here is error no available registry endpoint: failed to do request: . It means microk8s cannot find endpoint.

If you will run commend below, you will have the same output

$ cat /var/snap/microk8s/current/args/containerd-template.toml | grep endpoint

endpoint = ["https://registry-1.docker.io"]
endpoint = ["http://localhost:32000"]

As you see in microk8s endpoint is localhost not 127.0.0.1

Please change image in deployment-local.yaml

...spec:
  containers:
    - name: my-service-backend
      image: localhost/my-service
      imagePullPolicy: Always ...

Let me know if you will occur any other issues.

In my case I did not need tag and push image again with localhost, only change in YAML was required.

Answer 2

I also needed to comment out following line in the /etc/hosts (Ubuntu 18.04)

::1 localhost ip6-localhost

It seems that microk8s isn't capable of IPv6.