EC2 instance is out of service in loadbalancer

Question

I have an EC2 instance up and running. I have a load balancer where its associated with EC2 instance.

Ping Target         : HTTP:3001/healthCheck
Timeout             : 5 seconds
Interval            : 24 seconds
Unhealthy threshold : 2
Healthy threshold   : 10

Now the instance is shown as OutofService. I even tried changing listening ports and all. Things were working until,rebooted my EC2 instance. Any help would be higly appreciated.

Just for the info: I have rails app running at port 3001 and I have one listenser for HTTP:80(loadbalancer) to HTTP:3001.

I also have checked the working app through ssh in the terminal.

Answer 1

Suggestion#1:

If the current state of some or all your instances is OutOfService and the description field displays the message that the Instance has failed at least the Unhealthy Threshold number of health checks consecutively, the instances have failed the load balancer health check.

The following are the issues to look for, the potential causes, and the steps you can take to resolve the issues by following this link: Troubleshoot a Classic Load Balancer: Health Checks

Suggestion#2:

chrisa_pm has given some advice for this issue:

If you can confirm that your EC2 instance is reachable, you can remove it from your Load Balancer and add it back again. The Load Balancer will recognize it after a few minutes though.

Keep in mind that you need to confirm the health as it is set in your Health Check configuration:

1. For HTTP:80 you need to specify a page that is actually reachable (like index.html)
2. For TCP:80 it will only be needed access to the 80 TCP port.

Suggestion#3:

qh2 has make a solution by the following way

Create a service in startup to deregister and register again your instance.

Example: file awsloadbalancer

#!/bin/sh
chkconfig: 2345 95 20

When a isntance is stopped a load balancer is missed. this rebuild load balancer

case "$1" in
start)
aws --region eu-west-1 elb deregister-instances-from-load-balancer --load-balancer-name test --instances i-3c339b7c
aws --region eu-west-1 elb register-instances-with-load-balancer --load-balancer-name test --instances i-3c339b7c
;;
stop)
echo "stopping aws instances"
;;
restart)
echo "Restarting aws, nothing to do"
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
;;
esac

create file in /etc/init.d/ after that, register as service.

Suggestion#4:

Kenneth Snyder also solved the issue for specific ELB issue.

I also had similar issue but I was able to fix that.

I had created a security group for ELB which accepts request on port 80 and forward to EC2 on port 80. The security group that was earlier created for EC2 has also inbound rules for port 80 and RDP.

Still the instances were showing as OutOfService under ELB. Later i tried to add another inbound rule in the EC2's security group to allow port 80 for the SG that was created for ELB. and that worked.

I guess it requires the ELB SG to be allowed in the rules created for individual instance's SG. Hope that helps.

Resource Link:

https://forums.aws.amazon.com/thread.jspa?messageID=733153

Answer 2

Did you provide a health check endpoint and specified it in the EC2 console? Something like:

Note the port 80 and a valid route. You probably didn't set the port 3001 in your nginx / apache config

In the rails app, create an action like so:

class HealthCheckController < ActionController::Base
  def ping
    head :ok
  end
end

and route:

get 'health_check/ping'

The AWS load balancer will ping his endpoint and if the response is a 200 OK enough times (as per the Healthy threshold , it will deem the instance as "Healthy".

Answer 3

I see some issues with your ELB health check config. Right now, you have configured the health check to check an instance 10times every 24seconds before ELB will send requests. Hence, it takes

24seconds x 10 = 240secs # 4mins after reboot

assuming your Unicorn starts faster & doesn't die after its running, you should reduce the health check internal & healthy threshold.

• Reduce the Interval to 3-5seconds.
• Reduce Healthy threshold to 2-5times.

The above should help ELB to make instance "in service" faster.

This is assuming that your server config is properly setup to listen on /healthcheck port 3001 from external hosts. Please check your firewall/security groups/server config if that is not true.

Answer 4

The problem was, after rebooting the instance aws assigns new ip to the EC2 which I didn't notice.

And I was logging in thorough ssh to the old ec2 instance. And hence curl was never failing too.

(I am quite curious why this ip address still active and when I last checked it was active even after 15 days)

Nevertheless great check points(in general) by SkyWalker .

Finally what I had to:

With new ip, my pem file also got screwed up. Hence created new instance, new pem file, adjusted load balancer to point to this instance and security groups accorindgly.

PS:I couldn't be any more stupid.

Answer 5

确保appserver的安全组允许ELB安全组访问您在运行状况检查中指定的端口的运行状况检查端点。