Graceful Downscale GCE Instances

Question

I have a usecase in which i'm using GCE instances as NSQ consumers to work on async processes. This hostgroup have an autoscaling configured when the CPU is above 60% which usually gets triggered when there were alot of messages being consumed at that time. Following is the config:

However as per my understanding, once the average CPU goes below 60% for 15 seconds (based on the cooldown period), the instances will start to downscale and get terminated. Instant termination of the instance would cause the NSQ messages that were being consumed to be requeued(because of not sending the FIN command) causing a double process for the same message. Or sometimes the messages may even get lost.

I would like to know if there a way to maybe signal the instance that it is going to get downscaled? so there service could atleast do a graceful shutdown attempt.

Any other suggestions would be helpful as well

Answer 1

kudos to @JohnHanley for mentioning in the comment, this graceful shutdown is possible with the help of a shutdown script attached to the GCE Instance.

https://cloud.google.com/compute/docs/shutdownscript

Shutdown scripts are especially useful for VMs in a managed instance group with an autoscaler. If the autoscaler shuts down a VM in the group, the shutdown script runs before the VM stops and the shutdown script performs any actions that you define.

As stated in the docs, it has perfect use when your instance group gets autoscaled a lot. Inside the script, a simple systemctl stop would do the work to terminate the service binary