How to manage patching on multiple AWS accounts with different schedules

Question

I'm looking for the best way to manage patching Linux systems across AWS accounts with the following things to consider:

• Separate schedules to roll patches through Dev, QA, Staging and Prod sequentially
• Production patches to be released on approval, not automatic
• No newer patches can be deployed to Production than what was already deployed to lower environments (as new patches come out periodically throughout the month)

We have started by caching all patches in all environments on the first Sunday of every month. The goal there was to then install patches from cache. This helps prevent un-vetted patches being installed in prod.

Most, not all, instances are managed by OpsWorks, but there are numerous OpsWorks stacks. We have some other instances managed by Chef Server. Still others are not managed, but are just simple EC2 instances created from the EC2 console. This means, using recipes means we have to kick off approved patches on a stack-by-stack basis or instance-by-instance basis. Not optimal.

More recently, we have looked at the new features of SSM using a central AWS account to manage instances. However, this causes problems with some applications because the AssumeRole for SSM adds credentials to the .aws/config file that interferes with other tasks we need to run.

We have considered other tools, such as Ansible, but we would like to explore staying within the toolset we currently have which is largely OpsWorks and Chef Server. I'm looking for ideas that are more on a higher level, an architecture of how one would approach this scenario.

Thanks for any thoughts or ideas.

Answer 1

This sounds like one of the exact scenarios RunCommand was designed for.

You can create multiple groups of servers with different schedules based on tags. More importantly, you don't need to rely on secret/keys being deployed anywhere.