Tomcat restarts on AWS ELB

Question

How do people prevent Tomcat trying to serve traffic when it's restarting behind an Amazon Web Services Elastic Load Balancer?

Our applications take up to 40 seconds to restart once new software has been deployed to them. This is too small a window for the ELBs to notice, and so they Tomcat instance keep getting forwarded requests whilst they're waiting for their webapps to initialize, leading to users getting 500 error pages.

Is there a common way around this problem?

Answer 1

Franck is right, you should use the health check mechanism. However, the most frequent you can run that health check is every 6 seconds. That happens if in the mean time your app is restarted or redeployed? Unfortunately, the ELB doesn't know that, and keeps sending traffic to that instance. So in the worst case, during (about) 6 seconds your site's visitors that get served by that instance will get an error message or a trace dump.

I have contacted Amazon asking them for a PAUSE button for each instance that is enrolled with the ELB, so we could tell the ELB to stop sending request's to that instance. Someone there liked the idea, and promissed to escalate it to the developers. At least that would be an easy way to avoid that istance getting requests will it is not ready.

One advice not directly related to the ELB, but to Tomcat: Tomcat now has a way to upload WARs with versions. instead of uploading a "root.war" file, you can upload a "root##2012-04-30.war" file (for example), and Tomcat will deploy a new version of the same WAR. That new version won't interrupt the current version that is answering requests, and as soon as the new version is ready, it will start taking traffic instead of the old version. That way, your instance never stops answering requests, so the ELB doesn't need to stop sending requests there.

Answer 2

You can play with the Health Check parameter : Set up a special html page like /health-check.html in the Target parameter. When you want to shutdown an instance, remove/rename this page, then restart your tomcat and when it's up and running, rename the page to /health-check.html.