Azure Kubernetes 服务：如何将 Pod 从 Spot 节点池自动移动到常规节点池？

Question

I have 2 node pools in my azure kubernetes cluster. where one node pool is Spot VM node pool and another is a regular VM node pool. I have deployed 2 pods on the spot node pool. So, I want that if the spot node pool is get evicted then the pods on the same are to be rescheduled on regular node pool automatically?

I have learnt about node affinity and node selector which is used to run the pods in certain nodes. It would be helpful if kubernetes offer this feature to migrate the pods to another node automatically if the spot node pool/spot instances are get evicted.

Can any one know how can we achieve this in kubernetes?

Thanks.

Used Kubernetes version 1.18.14

Answer 1

You could use a NoSchedule taint on the spot nodes. That will not evict any running pods from the node, but it will not schedule any new pods to them (unless you specifically specify a matching toleration).

You can have a look at this documentation page for more details: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

But in general, what you do is the following:

There should be a unique label(s) on the spot nodes, you can find it if you describe one of the nodes. Use it to taint all the nodes with a NoSchedule taint like so

kubectl taint nodes node1 key1=value1:NoSchedule 

(replace the key1=value1 with the label you found)

For all pods you want to keep scheduling to the spot nodes (such as system pods) add the following toleration to allow them to continue:

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoSchedule"

For the pods you mentioned in your question, you don't want to reschedule them back to a spot node once it goes down, so just don't add the tolerations on them.

Assuming that your pods are controlled by a deployment or a stateful set (or any other controller that takes care to reschedule when it loses a pod), when your pods will get evicted from a spot node due it going away, the pods that will replace it will not be able to be scheduled to the spot nodes anymore, and given that the only other option is the regular nodes, they will be scheduled there.