为什么增加 GKE 集群的节点池大小后新节点实例不可用？

Question

I'm relatively new to K8's, and I've setup a private cluster with a single node pool on GKE. The node pool was configured to have nodes on 3 zones in a single GCP region (Autoscaling enabled with minimum of 2 nodes and maximum of 20 total nodes).

I ran into the following error upon trying to deploy new resources: Cannot schedule pods: Insufficient memory. I decided to disable Autoscaling on cluster's node pool, and then manually increase the number of nodes. I applied the change, each Instance Group now has 3 instances (a total of 9). However, the instances listed under the 'Nodes' section of the GKE Console does not reflect that. It show only 5 (which is actually a decrease from the original 6).

GCP DOCS: https://cloud.google.com/kubernetes-engine/docs/how-to/resizing-a-cluster

What am I missing here? Thanks

Answer 1

To clarify I am posting Community Wiki answer.

The problem here was caused by mismatch between versions of the cluster and node pool.

By default, automatic upgrades are enabled for Google Kubernetes Engine (GKE) clusters and node pools.

Here is documentation that helps to upgrade cluster or node pool manually.

---

Upgrading to the default cluster version can be done by running the following command:

gcloud container clusters upgrade CLUSTER_NAME --master

---

All nodes can be updated to the same version as the control plane by running this command:

gcloud container clusters upgrade

Following command should be used in case rolling back an upgrade:

gcloud container node-pools rollback NODE_POOL_NAME \  --cluster CLUSTER_NAME

---

See also this reference and this documentation .