如何在相对复杂的基础架构中正确自动扩展AWS EC2 Instances组？

Question

I'm working to migrate our servers on the Amazon Cloud, reasons are auto-scaling possibilities, costs, services, and more, obviously.

So far, I'm experimenting hard and trying to dive in the full-featured documentations, but with no previous experience I've much questions.

The envisaged infrastructure is the following:

                                  +-----+
                                  | ELB |
                                  +--+--+
                                     |
                +--------------------|--------------------+
                |            Auto-Scaling Group           |
                |--------------------|--------------------|
                |                    |                    |
                |  +---------+       |       +---------+  |
                |  | varnish |<------+------>| varnish |  |
                |  +----+----+               +---------+  |
                |       |                         |       |
                +-----------------------------------------+
                        |                         |
                        |                         |
                        |     +------------+      |
                        +---->|Internal ELB|<-----+
                              +------+-----+
                                     |
                +-----------------------------------------+
                |            Auto-Scaling Group           |
                |-----------------------------------------|
                |  +---------+       |       +---------+  |
                |  | Apache  |<------+------>| Apache  |  |
                |  +----+----+               +----+----+  |
                |       |                         |       |
                +-----------------------------------------+
                        |         +-----+         |
                        +-------->| RDS |<--------+
                                  +-----+

In words, I would have Elastic LoadBalancer which will send traffic to the Varnish instances, which will in turn send the traffic to an internal Elastic LoadBalancer which will send the traffic to the Apache frontends.

For now, I've discovered the AWS tools, like the CloudFormation service which seems able to bootstrap instance given a template, this seems great, but it seems able to bootstrap only.

Having a little experience with Puppet (and given the recommandation of AWS on the subject) I dove in the Puppet Master thing which is a great tool.

My idea, which may not be viable or realistic, is to create a "Puppet Node Stack" using CloudFormation templates, which will configure the instance as required and connect the puppet master to be provisioned.

Once I've a stack ready, I'm wondering how to configure/create Auto-Scaling group for both Varnish and Apache instances.

It seems that CFN has resources to configure the auto-scaling groups & policies, so I guess I could create two different templates for each.

But would the AS feature run trough the CFN service, and then does all the init things (and executes the user-data )?

I also read here and there that Puppet can make use of the EC2 Tags, maybe a generic stack template with corresponding tags (like roles) could do the trick?

Is this architecture realistic and viable? Do you have any feedbacks?

Thanks for your advices.

Answer 1

Auto-scaling creates new nodes based on the launch configuration. So you would have two separate auto scaling groups and two separate launch configurations. ie

"VarnishScalingGroup" : {
  "Type" : "AWS::AutoScaling::AutoScalingGroup",
  "Properties" : {
    "LaunchConfigurationName" : {"Ref" : "VarnishLaunchConfiguration" },
    "LoadBalancerNames" : {"Ref" : "ELB"},
    ...
  }
},
"VarnishLaunchConfiguration" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Properties" : {
    ...
    "UserData" : {
      ....
    },
    "MetaData" : {
      ...
    }
 },
"ApacheScalingGroup" : {
  "Type" : "AWS::AutoScaling::AutoScalingGroup",
  "Properties" : {
    "LaunchConfigurationName" : {"Ref" : "ApacheLaunchConfiguration" },
    "LoadBalancerNames" : {"Ref" : "InternalELB"},
    ...
  }
},
"ApacheLaunchConfiguration" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Properties" : {
    ...
    "UserData" : {
      ....
    },
    "MetaData" : {
      ...
    }
 }

The other thing you'd want to add is separate scaling policies for each scaling group, and appropriate CloudWatch metrics to match.

CloudFormation can also initiate updates to the stack. If as part of the userdata you kick of cfn-hup, then it will periodically (you decide) check for changes in the stack meta data - and then execute whatever you prefer. I tend to kick off another version of cfn-init - which will parse and update any meta data.

Key point - if you go down the cfn-hup path, it will not execute userdata again, unless the CloudFormation stack requires dropping and creating new instances.

One other point, if you want updates to the LaunchConfiguration to be rolled out, you need to ensure that the LaunchConfiguration also has an UpdatePolicy applied to it.

Answer 2

Instead of having a "Puppet Node Stack" you might want to consider pre-building your AMIs using a tool like packer ( http://www.packer.io/ ), which can provision a machine with puppet and create an AMI. Then add the provisioned AMI to your cloudformation template.

As Peter H. says, cloudformation can handle updates to your stack. So when you make changes to your puppet setup, you can build a new AMI and update your launch configuration in cloudformation. The autoscaling will start using the new AMI for autoscaling new instances.

Taking puppet out of cloudformation gives you a separation of concerns between infrastructure and server config.

Scaling up will happen faster with pre-built AMIs that already have your Apache/Varnish setup.

There are also advantages to a masterless puppet setup. ie. Decentralized, not having puppetmaster as a point of failure, etc. See https://serverfault.com/questions/408261/pros-and-cons-of-a-decentralized-puppet-architecture