Skaffold is not fully running all the services

Question

I'm developing this dummy project and trying to make it work locally via Skaffold .

There are 3 services in my project (running on ports 3001 , 3002 and 3003 respectively), wired via NATS server .

The problem is: I get different kinds of errors each time I run skaffold debug , and one/more service(s) don't work.

At times, I don't get any errors, and all services work as expected. The followings are some of the errors:

Waited for <...>s due to client-side throttling, not priority and fairness,
request: GET:https://kubernetes.docker.internal:6443/api/v1/namespaces/default/pods?labelSelector=app%!D(MISSING). <...>%!C(MISSING)app.kubernetes.io%!F(MISSING)managed-by%!D(MISSING)skaffold%!C(MISSING)skaffold.dev%!F(MISSING)run-id%!D(MISSING)<...>` (from `request.go:668`)
- `0/1 nodes are available: 1 Insufficient cpu.` (from deployments)
- `UnhandledPromiseRejectionWarning: NatsError: CONNECTION_REFUSED` (from apps)
- `UnhandledPromiseRejectionWarning: Error: getaddrinfo EAI_AGAIN nats-service` (from apps)

I'm at a loss and can't help myself anymore. I hope someone here will be able to help me out.

Thanks in advance.

PS: Below is my machine's config, in case it's my machine's fault.

Processor: AMD Ryzen 7 1700 (8C/16T)
Memory: 2 x 8GB DDR4 2667MHz
Graphics: AMD Radeon RX 590 (8GB)
OS: Windows 10 Pro 21H1

$ docker version
Client:
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.12
 Git commit:        0ed913b8-
 Built:             07/28/2020 16:36:03
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-11T18:10:22Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

I use WSL2 (Debian) and docker-desktop is the context of Kubernetes.

Answer 1

The error say it all:

- `0/1 nodes are available: 1 Insufficient cpu.` (from deployments)

You are requesting or deploying more apps than your node can handle.
In your deployment units ( Deployment , StatefulSet , DeamoSet etc) you define the resources which you require for your application, for example in this pod:

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: log-aggregator
    image: images.my-company.example/log-aggregator:v6
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

This is the relevant section:

resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Check your node resources and increase your nodes or resources in order to be able to deploy more services.

---

Answer 2

0/1 nodes are available: 1 Insufficient cpu. (from deployments)

This error clearly means your deployment doesn't have sufficient resources to startup.

i tried deploying the same on my environment same error, 2 CPU is not enough to start this image.

You might need a higher resource or big node to test

https://github.com/msrumon/microservice-template/blob/master/k8s/nats.yaml

Answer 3

The main reason of issues like this one is that you are setting only CPU limit (without setting CPU request) so Kubernetes automatically assigns a CPU request which is equal to the CPU limit :

If you specify a CPU limit for a Container but do not specify a CPU request, Kubernetes automatically assigns a CPU request that matches the limit. Similarly, if a Container specifies its own memory limit, but does not specify a memory request, Kubernetes automatically assigns a memory request that matches the limit.

So as requests are equal to the limits, your node can't meet these requirements (you have 16 CPUs available; to start all services you need 24 CPUs) - that's why you are getting 0/1 nodes are available: 1 Insufficient cpu error message.

How to fix it?

• Set a CPU request which is different from the CPU limit
• Delete CPU limits

But...

You wrote that:

Should I also try setting up the requests key and set the lower limit too? Or what about completely omitting it? I tried that one, and still same issue.

So if you deleted all CPU limits from all deployments and you still have error related to the insufficient CPU it clearly shows that your app is too resource-hungry. I'd suggest optimizing the application in terms of resource utilization. Another option is to increase node resources.