How can I add a health check to check if my instances in aws autoscale established a successful connection with another independent instance

Question

Currently I have multiple instances running under AWS autoscale group which polls messages from AWS SQS. My instances needs to establish connection with another instance before it start processing the incoming messages.

Sometimes it happens that one of my instances couldn't establish the connection, and I want to add a health check here to monitor the connection status and terminate the instance based on that. I don't think the default EC2 health check takes care of this scenario.

Is there any way I can add a health check to handle the above mentioned scenario.

Answer 1

You really need some sort of custom health check for this as any of the built in metrics (like CPU usage, network in/out) aren't going to work here.

One idea would be to have these instances bootstrapped with a cron job that runs every few minutes and checks if the the connection is up and running. If not, the instance could set itself as unhealthy using the CLI or SDK.

Here is an example piece of bash code that gets the instance ID of the instance the code is running on and then marks itself as unhealthy. This would trigger the ASG to replace it.

EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`"
aws autoscaling set-instance-health --instance-id $EC2_INSTANCE_ID --health-status Unhealthy

Resources:

• https://docs.aws.amazon.com/autoscaling/ec2/userguide/healthcheck.html