Kubernetes node CPU utilization

Question

I'm trying(learning) to figure out the best way to utilize CPU (and RAM) on k8s nodes . My final goal is to make sure CPU utilization on each node in the cluster is above X%

Till now I've read about cluster-autoscaler and HPA , but not sure if they'd help me with the use case.

From what I've read:

• cluster-autoscaler is used to autoscale nodes based on a comparison between replica count and resources.request Vs available CPU on the target ec2 instance - which is NOT based on traffic/actual CPU usage
• HPA is based on CPU/actual cpu usage, but for individual pods

I essentially wanna get to a point where kubectl top nodes would show all nodes are using > X% (let's say 60%) - and ideally trigger the autoscaling if we reach X2% (let's say 80%)

any suggestion/pointer on how to go about this use case? (or I should somehow use the combination of these 2 autoscaling mechanisms)

Answer 1

You can a combination of the HPA or/and Cluster autoscaler and/or the cloud providers' autoscaling group.

• HPA based on CPU/Memory of your pods and scale up and down your K8s Deployments for example.

• Cloud provider ASG or autoscaling group. Using the VMs or instances based and you can scale up and down based on their own CPU and memory metrics.

• Cluster autoscaler. It works when pods are pending and they have nowhere to run, but if you are handling the case above this is more of a safe fail mechanism or perhaps for workloads that don't require to come up very quickly.

In summary, you can use all 3 above (or less) but you have to see what works for you so that they don't conflict with each other. One potential problem is that when your cloud ASG starts scaling down then you also have pods in pending state then your cluster autoscaler (if you have it enabled) will kick in and you may have both of them trying to do the opposite causing your cluster to just not being able to schedule any pod.

✌️☮️