How do you put up a maintenance page for AWS when your instances are behind an ELB?

Question

How do you put up a maintenance page in AWS when you want to deploy new versions of your application behind an ELB? We want to have the ELB route traffic to the maintenance instance while the new auto-scaled instances are coming up, and only "flip over" to the new instances once they're fully up. We use auto-scaling to bring existing instances down and new instances, which have the new code, up.

The scenario we're trying to avoid is having the ELB serve both traffic to new EC2 instances while also serving up the maintenance page. Since we dont have sticky sessions enabled, we want to prevent the user from being flipped back and forth between the maintenance-mode page and the application deployed in an EC2 instance. We also can't just scale up (say from 2 to 4 instances and then back to 2) to introduce the new instances because the code changes might involve database changes which would be breaking changes for the old code.

Answer 1

The simplest way on AWS is to use Route 53 , their DNS service.

You can use the feature of Weighted Round Robin .

"You can use WRR to bring servers into production, perform A/B testing, or balance your traffic across regions or data centers of varying sizes."

More information in AWS documentations on this feature

EDIT: Route 53 recently added a new feature that allows DNS Failover to S3. Check their documentation for more details: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html

Answer 2

I realise this is an old question but after facing the same problem today (December 2018), it looks like there is another way to solve this problem.

Earlier this year, AWS introduced support for redirects and fixed responses to Application Load Balancers . In a nutshell:

• Locate your ELB in the console.
• View the rules for the appropriate listener.
• Add a fixed 503 response rule for your application's host name.
• Optionally provide a text/plain or text/html response (ie your maintenance page HTML).
• Save changes.

Once the rule propagates to the ELB (took ~30 seconds for me), when you try to visit your host in your browser, you'll be shown the 503 maintenance page.

When your deployment completes, simply remove the rule you added.

Answer 3

Route53 is not a good solution for this problem. It takes a significant amount of time for DNS entries to expire before the maintenance page shows up (and then it takes that same amount of time before they update after maintenance is complete). I realize that Lambda and CodeDeploy triggers did not exist at the time this question was asked, but I wanted to let others know that Lambda can be used to create a relatively clean solution for this, which I have detailed in a blog post: http://blog.ajhodges.com/2016/04/aws-lambda-setting-temporary.html

The jist of the solution is to subscribe a Lambda function to CodeDeploy events, which replaces your ASG with a micro instance serving a static page in your load balancer during deployments.

Answer 4

Came up with another solution that's working great for us. Here are the required steps to get a simple 503 http response:

1. Replicate your EB environment to create another one, call it something like app-environment-maintenance , for instance.
2. Change the configuration for autoscaling and set the min and max servers both to zero. This won't cost you any EC2 servers and the environment will turn grey and sit in your list.

3. Finally, you can use the AWS CLI to now swap the environment CNAME to take your main environment into maintenance mode. For instance:

    aws elasticbeanstalk swap-environment-cnames \\ --profile "$awsProfile" \\ --region "$awsRegion" \\ --output text \\ --source-environment-name app-prod \\ --destination-environment-name app-prod-maintenance 

This would swap your app-prod environment into maintenance mode. It would cause the ELB to throw a 503 since there aren't any running EC2 instances and then Cloudfront can catch the 503 and return your custom 503 error page, should you wish, as described below.

---

Bonus configuration for custom error pages using Cloudfront:

We use Cloudfront, as many people will for HTTPS, etc. Cloudfront has error pages. This is a requirement.

1. Create a new S3 website hosting bucket with your error pages. Consider creating separate files for response codes, 503, etc. See #6 for directory requirements and routes.
2. Add the S3 bucket to your Cloudfront distribution.
3. Add a new behavior to your Cloudfront distribution for a route like /error/* .
4. Setup an error pages in Cloudfront to handle 503 response codes and point it to your S3 bucket route, like /error/503-error.html

Now, when your ELB thorws a 503, your custom error page will be displayed.

And that's it. I know there are quite a few steps to get the custom error pages and I tried a lot of the suggested options out there including Route53, etc. But all of these have issues with how they work with ELBs and Cloudfront, etc.

Note that after you swap the hostnames for the environments, it takes about a minute or so to propagate.

Answer 5

Our deployment process first runs a cloudformation to spun up a ec2 micro instance (Maintenance instance) which copies pre-defined static page from s3 onto the ec2. Cloudformation is supplied with elb's to which micro ec2 instance is attached. Then a script (powershell or cli) is run to remove web instances (ec2) from elb's leaving Maintenance instance.

This way we switch to maintenance instance during deployment process.

In our case, we have two elb's, one for external and the other internal. Our internal elb's will not be updated during this process and is how we have post prod deployment smoke test is done. Once testing is done, we run another script to attach web instances back to elb's and delete the Maintenance stack.

Answer 6

As far as I could see, we were in a situation where the above answers didn't apply or weren't ideal.

We have a Rails application running the Puma with Ruby 2.3 running on 64bit Amazon Linux/2.9.0 that seems to come with a (classic) ELB.

So ALB 503 handling wasn't an option.

We also have a variety hardware clients that I wouldn't trust to always respect DNS TTL, so Route53 is risky.

What did seem to work nicely is a secondary port on the nginx that comes with the platform.

I added this as .ebextensions/maintenance.config

files:
  "/etc/nginx/conf.d/maintenance.conf":
    content: |
      server {
        listen 81;
        server_name _ localhost;
        root /var/app/current/public/maintenance;
      }

container_commands:
  restart_nginx:
    command: service nginx restart

And dropped a copy of https://gist.github.com/pitch-gist/2999707 into public/maintenance/index.html

Now to set maintenance I just switch my ELB listeners to point to port 81 instead of the default 80. No extra instances, s3 buckets or waiting for clients to fresh DNS.

Only takes maybe ~15s or so for beanstalk (probably mostly waiting for cloudformation in the back-end) to apply.