Unable to launch new pods despite resources seemingly available

Question

I'm unable to launch new pods despite resources seemingly being available.

Judging from the below screenshot there should be room for about 40 new pods.

And also judging from the following screenshot the nodes seems fairly underutilized

However I'm currently facing the below error message

0/3 nodes are available: 1 Insufficient cpu, 2 node(s) had volume node affinity conflict.

And last night it was the following

0/3 nodes are available: 1 Too many pods, 2 node(s) had volume node affinity conflict.

Most of my services require very little memory and cpu. And therefore their resources are configured as seen below

resources:
  limits:
    cpu: 100m
    memory: 64Mi
  requests:
    cpu: 100m
    memory: 32Mi

Why I can't deploy more pods? And how to fix this?

Answer 1

Your problem is "volume node affinity conflict".

From Kube.netes Pod Warning: 1 node(s) had volume node affinity conflict :

The error "volume node affinity conflict" happens when the persistent volume claims that the pod is using are scheduled on different zones, rather than on one zone, and so the actual pod was not able to be scheduled because it cannot connect to the volume from another zone.

First, try to investigate exactly where the problem is. You can find a detailed guide here . You will need commands like:

kubectl get pv  
kubectl describe pv  
kubectl get pvc  
kubectl describe pvc

Then you can delete the PV and PVC and move pods to the same zone along with the PV and PVC.

Answer 2

volume node affinity conflict - the volume you tried to mount is not available on any of the node. You can resolve this or paste your volumes section to the question for further examination.