Kubernetes 中 Pod 保证的 QoS 等级

Question

On my kubernetes nodes there are

1. prioritized pods
2. dispensable pods

Therefore I would like to have QoS class of Guaranteed for the prioritized pods. To achieve a Guaranteed class the cpu/memory requests/limits must meet some conditions . Therefore:

For every Container in the Pod, the CPU limit must equal the CPU request

But I would like to set a higher CPU limit than request, so that the prioritized pods can use every free CPU resources which are available.

Simple example: A Node with 4 cores has:

• 1 prioritized pod with 2000 CPU request and 3900 CPU limit
• 3 dispensable pods with each 500 CPU request and limit.

If the prioritized pod would have 2000 CPU request and limit 2 Cores are wasted because the dispensable pods don't use CPU most of the time.

If the prioritized pod would have 3900 CPU request and limit, I would need an extra node for the dispensable pods.

Questions

Is it possible to set explicitly the Guaranteed class to a pod even with difference CPU request and limit?

If it's not possible: Why is there no way to explicitly set the QoS class?

Remarks

There's an system-cluster-critical option. But I think this should only be used for critical k8s add-on pods but not for critical applications.

Answer 1

Is it possible to set explicitly the Guaranteed class to a pod even with difference CPU request and limit?

Yes, however you will need to use an additional plugin: capacity-scheduling used with PriorityClass :

There is increasing demand to use Kubernetes to manage batch workloads (ML/DL). In those cases, one challenge is to improve cluster utilization while ensuring that each user has a reasonable amount of resources. The problem can be partially addressed by the Kubernetes ResourceQuota . The native Kubernetes ResourceQuota API can be used to specify the maximum overall resource allocation per namespace. The quota enforcement is done through an admission check. A quota consumer (eg, a Pod) cannot be created if the aggregated resource allocation exceeds the quota limit. In other words, the overall resource usage is aggregated based on Pod's spec (ie, cpu/mem requests) when it's created. The Kubernetes quota design has the limitation: the quota resource usage is aggregated based on the resource configurations (eg, Pod cpu/mem requests specified in the Pod spec). Although this mechanism can guarantee that the actual resource consumption will never exceed the ResourceQuota limit, it might lead to low resource utilization as some pods may have claimed the resources but failed to be scheduled. For instance, actual resource consumption may be much smaller than the limit.

---

Pods can be created at a specific priority . You can control a pod's consumption of system resources based on a pod's priority, by using the scopeSelector field in the quota spec.

A quota is matched and consumed only if scopeSelector in the quota spec selects the pod.

When quota is scoped for priority class using scopeSelector field, quota object is restricted to track only following resources:

• pods
• cpu
• memory
• ephemeral-storage
• limits.cpu
• limits.memory
• limits.ephemeral-storage
• requests.cpu
• requests.memory
• requests.ephemeral-storage

This plugin supports also preemption (example for Elastic):

Preemption happens when a pod is unschedulable, ie, failed in PreFilter or Filter phases.

In particular for capacity scheduling, the failure reasons could be:

• Prefilter Stage
• sum(allocated res of pods in the same elasticquota) + pod.request > elasticquota.spec.max
• sum(allocated res of pods in the same elasticquota) + pod.request > sum(elasticquota.spec.min)

So the preemption logic will attempt to make the pod schedulable, with a cost of preempting other running pods.

Examples of yaml files and usage can be found in the plugin description .