Kube.netes Pod Scheduling failed with 1 node(s) didn't match pod affinity/anti-affinity

Question

We have got a Kube.netes cluster up and running (with 1 master and 1 slave).

Recently we added a new worker node (temp-worker) which was successful and all were on Cent OS 8

Node Details

NAME            STATUS   ROLES                  AGE     VERSION
master-node     Ready    control-plane,master   34d     v1.20.2
temp-worker     Ready    <none>                 5h48m   v1.20.2
worker-node-2   Ready    worker                 34d     v1.20.0

I tried with the following pod specification (with 2 replicas), however pod remained in state Pending

Pod Spec

   podAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
     - labelSelector:
         matchExpressions:
         - key: app                      #security
           operator: DoesNotExist
       topologyKey: kubernetes.io/hostname
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app                      #security
              operator: In
              values:
              - hello-app
          topologyKey: kubernetes.io/hostname

Pod logs

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  3m50s  default-scheduler  0/3 nodes are available: 1 node(s) didn't match pod affinity rules, 1 node(s) didn't match pod affinity/anti-affinity, 2 node(s) had volume node affinity conflict.
  Warning  FailedScheduling  3m50s  default-scheduler  0/3 nodes are available: 1 node(s) didn't match pod affinity rules, 1 node(s) didn't match pod affinity/anti-affinity, 2 node(s) had volume node affinity conflict.

I tried by changing the operator values (eg In, NotIn etc) in pod specification but nothing changed

I would like to understand why it reports error for the newly added worker-node. What did I do wrong?

Answer 1

From this kube.netes doc “The Deployment controller stops the bad rollout automatically, and stops scaling up the new ReplicaSet. This depends on the rollingUpdate parameters (maxUnavailable specifically) that you have specified. Kube.netes by default sets the value to 25%.”

Means that it prevents Kube.netes from evicting one pod out of three. You are attempting a deployment update.

Try adding the Max unavailable to deployment.

strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1

As per this doc check whether all conditions are satisfied in your pod affinity.

Try specifying the label values in the podAffinity, Maybe there are no matching nodes available in the that label values follow this doc for more information.