扩展 Kube.netes 集群以处理队列中的作业？

Question

(I'm new to Kube.netes and not sure this is best practice)

I have a pipeline of jobs in a Firestore database that need to be completed as quickly as possible.

I want to set up a Kube.netes cluster (on GKE) that will scale up when there is a large backlog of tasks to complete. Each pod/node needs a single GPU to complete the task.

Is it possible to use a cloud function to manually scale the number of nodes depending on the number of jobs in the pipeline?

I was planning on using the clusters.nodePools.setSize() function from the GKE client library but I'm not sure if this is just intended for initial cluster setup rather than manual scaling.

Thanks

https://cloud.google.com/kube.netes-engine/docs/reference/rest/v1beta1/projects.locations.clusters.nodePools/setSize

Answer 1

You can configure and use Horizontal pod scaling on your cluster to scale the number of Pods.

As suggested by @somethingsomething refer these links on Horizontal Pod Autoscaler and Cluster autoscaler :

The Horizontal Pod Autoscaler changes the shape of your Kube.netes workload by automatically increasing or decreasing the number of Pods in response to the workload's CPU or memory consumption, or in response to custom metrics reported from within Kube.netes or external metrics from sources outside of your cluster.

Horizontal Pod autoscaling helps to ensure that your workload functions consistently in different situations, and allows you to control costs by only paying for extra capacity when you need it.

It's not always easy to predict the indicators that show whether your workload is under-resourced or under-utilized. The Horizontal Pod Autoscaler can automatically scale the number of Pods in your workload based on one or more metrics.