How to resize K8s cluster with kops, cluster-autoscaler to dynamically increase Masters

Question

We have configured Kubernetes cluster on EC2 machines in our AWS account using kops tool ( https://github.com/kubernetes/kops ) and based on AWS posts ( https://aws.amazon.com/blogs/compute/kubernetes-clusters-aws-kops/ ) as well as other resources.

We want to setup a K8s cluster of master and slaves such that:

1. It will automatically resize (both masters as well as nodes/slaves) based on system load.
2. Runs in Multi-AZ mode ie at least one master and one slave in every AZ (availability zone) in the same region for eg us-east-1a, us-east-1b, us-east-1c and so on.

We tried to configure the cluster in the following ways to achieve the above.

1. Created K8s cluster on AWS EC2 machines using kops this below configuration: node count=3, master count=3, zones=us-east-1c, us-east-1b, us-east-1a. We observed that a K8s cluster was created with 3 Master & 3 Slave Nodes. Each of the master and slave server was in each of the 3 AZ's.

2. Then we tried to resize the Nodes/slaves in the cluster using ( https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml ). We set the node_asg_min to 3 and node_asg_max to 5. When we increased the workload on the slaves such that auto scale policy was triggered, we saw that additional (after the default 3 created during setup) slave nodes were spawned, and they did join the cluster in various AZ's. This worked as expected. There is no question here.

3. We also wanted to set up the cluster such that the number of masters increases based on system load. Is there some way to achieve this? We tried a couple of approaches and results are shared below:

A) We were not sure if the cluster-auto scaler helps here, but nevertheless tried to resize the Masters in the cluster using ( https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml ). This is useful while creating a new cluster but was not useful to resize the number of masters in an existing cluster. We did not find a parameter to specify node_asg_min, node_asg_max for Master the way it is present for slave Nodes. Is there some way to achieve this?

B) We increased the count MIN from 1 to 3 in ASG (auto-scaling group), associated with one the three IG (instance group) for each master. We found that new instances were created. However, they did not join the master cluster. Is there some way to achieve this?

Could you please point us to steps, resources on how to do this correctly so that we could configure the number of masters to automatically resize based on system load and is in Multi-AZ mode?

Kind regards, Shashi

Answer 1

There is no need to scale Master nodes.

Master components provide the cluster's control plane. Master components make global decisions about the cluster (for example, scheduling), and detecting and responding to cluster events (starting up a new pod when a replication controller's 'replicas' field is unsatisfied).

Master components can be run on any machine in the cluster. However, for simplicity, set up scripts typically start all master components on the same machine, and do not run user containers on this machine. See Building High-Availability Clusters for an example multi-master-VM setup.

Master node consists of the following components:

kube-apiserver

Component on the master that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane.

etcd

Consistent and highly-available key value store used as Kubernetes' backing store for all cluster data.

kube-scheduler

Component on the master that watches newly created pods that have no node assigned, and selects a node for them to run on.

kube-controller-manager

Component on the master that runs controllers .

cloud-controller-manager

runs controllers that interact with the underlying cloud providers. The cloud-controller-manager binary is an alpha feature introduced in Kubernetes release 1.6.

For more detailed explanation please read the Kubernetes Components docs. Also if You are thinking about HA, you can read about Creating Highly Available Clusters with kubeadm

Answer 2

I think your assumption is that similar to kubernetes nodes, masters devide the work between eachother. That is not the case, because the main tasks of masters is to have consensus between each other. This is done with etcd which is a distributed key value store. The problem maintaining such a store is easy for 1 machine but gets harder the more machines you add.

The advantage of adding masters is being able to survive more master failures at the cost of having to make all masters fatter (more CPU/RAM....) so that they perform well enough.