How to autoscale Cassandra cluster on AWS or how to node dynamically to the cluster depending upon the load

Question

I have been trying to auto-scale a 3node Cassandra cluster with Replication Factor 3 and Consistency Level 1 on Amazon EC2 instances.

What steps do I need to perform to add/remove nodes to the cluser dynamically based on load on application?

Answer 1

Unfortunately scaling up and down responding to the current load is not straightforward, and if you have a cluster with a large amount of data, this won't be possible:

• you can't add multiple nodes simultaneously to a cluster, all the operations need to be sequential.
• adding or removing a node will require to stream data in or out of the node; this will depend on the size of your data, as well as the EC2 instance type you are using (for the network bandwidth limit); also, there will be differences if you are utilizing instance storage or EBS (EBS will limit you in IOPS)
• You mentioned that you are using AWS and a replication factor of 3, are you also using different availability zones (AZ's)? if you are, the EC2Snitch will work to ensure that the information is balanced between them, in order to be resilient, when you are scaling up and down you will need to keep an even distribution between AZ's.
• The scale operations will cause a rearrangement in the distribution of tokens, once that it is completed you will need to do a cleanup ( nodetool cleanup ) to remove data that is not in use anymore by the node; this operation will also take time. This is important to keep in mind if you are scaling up because you are running out space.

For our use case, we are getting good results taking a proactive approach, we have set up an aggressive alert/monitoring strategy to have an early detection, so we can start the scale up operations before there is any performance impact. If your application or use case has a predictable pattern of usage can also help you to take actions in preparation of periods of high workloads.