如何处理 AWS elasticbeanstalk 上的金丝雀发布？

Question

I have previously seen it done by having one EC2 instance running HAProxy, configured via a json file/lambda function, that in turn controlled the traffic with sticky sessions, into two separate elasticbeanstalk applications. So we have two layers of load balancing.

However, this has a few issues, one being: Testing several releases becomes expensive, requires more and more EB applications.

By canary release, I mean, being able to release to only a percentage of traffic, to figure out any errors that escaped the devs, the review process, and the QA process, without affecting all traffic.

What would be the best way to handle such a setup with AWS resources and not break the bank? :)

Answer 1

I found this Medium article that explain the usage of passive autoscaling group where you deploy the canary version into it and monitor for statistics. Once you are satisfied with the result, you can change the desired count for the canary autoscaling group to 0, and perform rolling upgrade to the active autoscaling group.

Here is the link to the article: https://engineering.klarna.com/simple-canary-releases-in-aws-how-and-why-bf051a47fb3f

Answer 2

The way you would achieve canary testing with elastic beanstalk is by

1. Create a 2nd beanstalk environment to which you deploy the canary release
2. Use a Route53 Weighted routing policy to send a percentage of the DNS requests to your canary environment.
3. If you're happy with the performance of the canary you can then route 100% of the traffic to the canary env, etc.

Something to keep in mind with DNS routing is, that the weighted routing is not an exact science since clients cache DNS based on the TTL you set in Route53. In the extreme scenario where you would have eg only one single client calling your beanstalk environment (such as aa single web server) and the TTL is set to 5 minutes, it could happen that the switching between environments only happens every 5 minutes.

Therefore for weighted routing it is recommended to use a fairly low TTL value. Additionally having many clients (eg mobile phones) works better in conjunction with DNS routing.

Alternatively it might be possible to create a separate LB in front of the two beanstalk environments that balances requests between the beanstalk environments. However I'm not 100% sure if a LB can sit in front other (beanstalk) LBs. I suspect the answer is not but I haven tried yet.

Modifying the autoscaling group in elastic beanstalk is not possible, since the LB is managed by beanstalk and beanstalk can decide to revert the changes you did manually on the LB. Additionally beanstalk does not allow you to deploy to a subset of instances while keeping the older version on another subset.

Hope this helps.

Answer 3

Traffic splitting is supported natively by Elastic Beanstalk.

Be sure to select a "high availability" config preset when creating your application environment (by clicking on "configure more options"), as this will configure a load balancer for your env:

Then edit the "Rolling updates and deployments" section of your environment and choose "Traffic splitting" as your deployment strategy.