Role Switching ( assume role ) with AWS and Terraform

Question

Background:

We've started a spike investigation into generating infrastructure using Terraform rather than directly with Cloudformation.

We have multiple AWS accounts which are separated for Live, QA and Dev environments ( complete separation of concerns due to the complexity of stacks and the potential for catastrophic destruction of client services ). Our accounts have MFA switched on.

With Cloudformation we undertake role switching to authenticate against one primary AWS account and then stand up our stacks in the correct account using an assumed role.

The crux of the question:

Is this possible ( without massive dirty hacks, please! ) in Terraform? We have been attempting this process but suffer the following error when trying to run Terraform Plan or Build

" The role ' arn:aws:iam::ACCOUNTID:role/ASSUMEDROLE" cannot be assumed.'

Our provider switching code is:

# Configure the AWS Provider
provider "aws" {
  region = "${var.aws_region}"
  profile = "${var.profile}"
  assume_role {
    role_arn = "arn:aws:iam::${lookup(var.aws_account_id, var.tag_environment)}:role/MYASSUMEROLE"
  }
}

From hours of Googling, reading through blog posts and Terraform's open bug list, this seems to be something that isn't supported yet?

We've seen that at least one person is creating shell scripts to try and do the authentication and then pass-through. This seems a really ugly hack to make it work.

Has anyone actually got this working with MFA turned on with the accounts?

We've had extremely vague responses from the team at HashiCorp when talking at Cons and workshops.

Answer 1

I manage an AWS organization that has over 100 accounts. Everyone has a single IAM user in an account we call identity . Then they sts:AssumeRole to IAM roles in other accounts that have a trust relationship naming the identity account as trusted. Users are responsible for running the script I'm providing to generate MFA aws config profiles. The terraform itself isn't doing this because there is a need to enter a manual code.

tips on setting up the roles

Make IAM groups in identity, and give them permission to assume the corresponding roles in desired accounts. Make sure to also give permissions for the user to be able to self manage passwords and MFA settings in the identity account. Make sure there is not an MFA condition on the self-manage permissions because they can't add an MFA device if they don't have permission due to the condition. It's a chicken and egg issue. After setting up MFA, people will need to log out and back in with MFA to satisfy MFA conditions on IAM policies.

When you make the roles in other accounts, you have to create a trust policy to trust the identity account. When you do this, I recommend adding the following condition to true: MultiFactorAuthPresent .

setting up the aws config and credentials files

My recommendation is to make a pattern of profile names that must be set within your organization. You can have many, many profiles in your config. I have hundreds. They are generated, not manually maintained.

[org]
aws_access_key_id     = SomeKey
aws_secret_access_key = SomeSecretKey

aws configure set profile.org.username gmiller.cli

[profile org]
region   = us-west-2
username = jsmith
roles    = admin,read,terraform
accounts          = identity,shared_services,dev_a,dev_b,dev_c,uat_a,uat_b,uat_c
account_numbers   = 
  identity        = 566179001270272
  shared_services = 886917640172339
  dev_a           = 505685932297420
  dev_b           = 488489750836019
  dev_c           = 695182558652006
  uat_a           = 123189319014809
  uat_b           = 705170270846976
  uat_c           = 608206892249907

generating mfa profiles via script

My script works by using your non-MFA AccessKey and SecretAccessKey to request MFA backed auth keys. To do this, you call the mfa commands in the aws cli and pass the current MFA code. My script then parses the return body and creates a new profile with _mfa added to the end of the original profile name. So any time you want to use profile foo but it needs to be MFA, just specify profile foo_mfa . If you get a message saying that they keys are expired, you need to run the script again.

A note about the script, I have since reworked this into a much better version in golang. But it's mixed with stuff I don't want to share yet and maybe some day I'll release that part when I clean it up. This is my first version written in bash. It does just fine. It also rotate your key in the profile you specify. It makes a new key, updates your profile to use the new keys. Then it deletes your old key. It does this on every execution. So this script also rotates your keys so you don't have to remember to or get locked out due to organization policy.

The script also generates all your other policies for you. You can list all the accounts and role combinations you want profiles for. Then you have to put the account number in the map account_numbers

Don't forget you can use commands like configure get profile.cde.account_numbers.identity 566179001270272 to setup the config. I also like to put this scipt in the ~/.aws directory alongside all the other AWS config.

run: ~/.aws/mfa.sh --realm org --code 729376

From your source profile, org , this will generate the following:

[org_mfa]
aws_access_key_id     = KeyThatWillExpire
aws_secret_access_key = SecretKeyThatWillExpire
aws_session_token     = SessionTokenThatWillExpire/////////////gornucibawowovvawumekuvekorsekotworwatandencitezesodupusowoimmelavdufzocpunbofubafdofizagvuchecufihencehfejjehdaakacmudkiutmotuwwomcoejbokazejudocetbovmifwavawvilidmalwermizmurtutotabujobgajpihsoticoowitoicubukbuglahicpatjuswodiklawciredemkukudapafietwepophibtetdildewdivwizhadunantizozatohojasejorjeivirurenmajrudsopujkalahoidugacsogogojwaprildibovgabzirajimwegegupnidukogafupaniwutudtiruntuzsogucopawafuvudfimozasbitokpulduhwagjubbevamatuopijogihaj

You can check to see if it worked with a command like: aws --profile=org_mfa sts get-caller-identity

You can then make all your other profiles expect org_mfa to exist. This is useful for running cli commands but for terraform look below. Profiles generated by my script will automatically do this for you.

[profile org_some_account_terraform]
source_profile = org_mfa
role_arn       = arn:aws:iam::123otheraccount321:role/terraform
region         = us-west-2
output         = json

In Terraform, you can use variables for the profile and assume_role properties. This is where having a standard pattern for role naming in your organization pays off. Don't have people pass in the profile they want to use, dictate that in the terraform code and have your users create profiles that match code expectations. I don't get complaints about this. It makes life super easy.

Terraform provider with the MFA role specified:

provider "aws" {
  version = "~> 2.38.0"
  alias   = "shared_services"
  profile = format("%s_mfa", var.realm)
  region  = var.region

  assume_role {
    role_arn = "arn:aws:iam::${var.shared_services_account_number}:role/terraform"
  }
}

This provider establishes a session for aws resource creation in my account I call shared_services . It does so using the profile generated by the mfa script via my org profile that has my user's access key and secret access keys.

Then, take advantage of provider mapping to pass specific providers to specific modules, if you need to. See the providers mapping below:

module "bootstrap" {
  source = "../_modules/bootstrap/global"

  providers = {
    aws                 = aws
    aws.org_identity    = aws.org_identity
    aws.shared_services = aws.shared_services
  }

  iam_alias = var.iam_alias
  realm     = var.realm
}

I've been running this setup for at least 2 years. It's worked with no disappointment or issues. I hope this answers your question. My script is below:

#!/usr/bin/env bash

# TODO generate config and credentials from gomplate
# TODO test each role assumption to validate config vs reality

# https://natelandau.com/boilerplate-shell-script-template/
# ##################################################
# My Generic BASH script template
#
version="1.0.0"               # Sets version variable
#
scriptTemplateVersion="1.3.0" # Version of scriptTemplate.sh that this script is based on
#                               v.1.1.0 - Added 'debug' option
#                               v.1.1.1 - Moved all shared variables to Utils
#                                       - Added $PASS variable when -p is passed
#                               v.1.2.0 - Added 'checkDependencies' function to ensure needed
#                                         Bash packages are installed prior to execution
#                               v.1.3.0 - Can now pass CLI without an option to $args
#
# HISTORY:
#
# * DATE - v1.0.0  - First Creation
#
# ##################################################

# Provide a variable with the location of this script.
scriptPath="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
scriptParentPath="${scriptPath%/*}"

# Source Scripting Utilities
# -----------------------------------
# These shared utilities provide many functions which are needed to provide
# the functionality in this boilerplate. This script will fail if they can
# not be found.
# -----------------------------------

# utilsLocation="${scriptParentPath}/lib/utils.sh" # Update this path to find the utilities.

# if [ -f "${utilsLocation}" ]; then
#   source "${utilsLocation}"
# else
#   echo "Please find the file util.sh and add a reference to it in this script. Exiting."
#   exit 1
# fi

# trapCleanup Function
# -----------------------------------
# Any actions that should be taken if the script is prematurely
# exited.  Always call this function at the top of your script.
# -----------------------------------
# function trapCleanup() {
#   echo ""
#   if is_dir "${tmpDir}"; then
#     rm -r "${tmpDir}"
#   fi
#   die "Exit trapped."  # Edit this if you like.
# }

# Set Flags
# -----------------------------------
# Flags which can be overridden by user input.
# Default values are below
# -----------------------------------
quiet=0
printLog=0
verbose=0
force=0
strict=0
debug=0
args=()

# args
code=""
realm=""
region="us-west-2"
mfa_arn=""
username=""
account_number=""
skip_key_rotate=0
skip_realm_config=0
duration_seconds=129600

# scratch vars
exit_do_to_missing_required_vars=0
return_body=""
aws_session_token=""
secret_access_key=""
access_key_id=""
old_key_id=""
new_key_id=""
old_secret=""
new_secret=""
declare -a accounts
declare -a roles

# Set Temp Directory
# -----------------------------------
# Create temp directory with three random numbers and the process ID
# in the name.  This directory is removed automatically at exit.
# -----------------------------------
tmpDir="/tmp/${scriptName}.$RANDOM.$RANDOM.$RANDOM.$$"
(umask 077 && mkdir "${tmpDir}") || {
  echo "Could not create temporary directory! Exiting."
  exit 1
}

# Logging
# -----------------------------------
# Log is only used when the '-l' flag is set.
#
# To never save a logfile change variable to '/dev/null'
# Save to Desktop use: $HOME/Desktop/${scriptBasename}.log
# Save to standard user log location use: $HOME/Library/Logs/${scriptBasename}.log
# -----------------------------------
logFile="$HOME/Library/Logs/${scriptBasename}.log"

# Check for Dependencies
# -----------------------------------
# Arrays containing package dependencies needed to execute this script.
# The script will fail if dependencies are not installed.  For Mac users,
# most dependencies can be installed automatically using the package
# manager 'Homebrew'.
# -----------------------------------
homebrewDependencies=()

function verbose() {
  if [[ $verbose -eq 1 ]]; then
    echo $1
  fi
}

function mainScript() {
  ############## Begin Script Here ###################
  ####################################################

  echo -n
  verbose "starting script"

  verbose "checking if required code param is set"
  if [[ $code == "" ]]; then
    verbose "exiting because required code param isn't set"
    echo "code or c is required"
    exit_do_to_missing_required_vars=1
  fi
  verbose "code param is set to ${code}"

  verbose "checking if required realm param is set"
  if [[ $realm == "" ]]; then
    verbose "exiting because required code param isn't set"
    echo "realm or r is required"
    exit_do_to_missing_required_vars=1
  fi
  verbose "realm param is set to ${realm}"

  verbose "checking to see if exit_do_to_missing_required_vars is 1"
  if [[ $exit_do_to_missing_required_vars -eq 1 ]]; then
    verbose "exit_do_to_missing_required_vars is 1 so exiting..."
    usage
    exit
  fi
  verbose "exit_do_to_missing_required_vars is not 1"

  verbose "setting region to: ${region}"
  region=$region
  aws configure set profile.${realm}.region $region

  verbose "setting username var: aws configure get username --profile $realm"
  username=$(aws configure get username --profile $realm)
  verbose "username is set to: ${username}"
  verbose "checking account number"
  account_number=$(aws configure get account_numbers.identity --profile $realm)
  verbose "account number is set to: ${account_number}"

  verbose "checking if required username aws config is set"
  if [[ $username == "" ]]; then
    verbose "exiting because required username aws config isn't set"
    echo "username is required to be set your realm's .aws/credentials profile"
    exit_do_to_missing_required_vars=1
  fi

  verbose "checking if required accounts and account_numbers aws config is set"
  if [[ $account_number == "" ]]; then
    verbose "exiting because required accounts and account_numbers aws config isn't set"
    echo "account_number is required to be set your realm's .aws/credentials profile"
    exit_do_to_missing_required_vars=1
  fi

  verbose "checking to see if exit_do_to_missing_required_vars is 1"
  if [[ $exit_do_to_missing_required_vars -eq 1 ]]; then
    verbose "exit_do_to_missing_required_vars is 1 so exiting..."
    usage
    exit
  fi

  verbose "creating MFA arn from account number and username"
  mfa_arn=arn:aws:iam::${account_number}:mfa/${username}
  verbose "mfa_arn = ${mfa_arn}"

  verbose "getting session token body by executing:"
  verbose "shell aws --profile=$realm sts get-session-token --serial-number $mfa_arn --token-code $code --duration-seconds $duration_seconds"
  return_body=$(aws --profile=$realm --region=$region sts get-session-token --serial-number $mfa_arn --token-code $code --duration-seconds $duration_seconds)
  verbose "session token body ="
  verbose $return_body
  verbose "getting keys from body"
  aws_session_token=$(echo $return_body | jq -r '.Credentials | .SessionToken')
  verbose "aws_session_token = ${aws_session_token}"
  secret_access_key=$(echo $return_body | jq -r '.Credentials | .SecretAccessKey')
  verbose "secret_access_key = ${secret_access_key}"
  access_key_id=$(echo $return_body | jq -r '.Credentials | .AccessKeyId')
  verbose "access_key_id = ${access_key_id}"

  if [[ $skip_key_rotate -eq 0 ]]; then
    verbose "skip key rotation not enabled: rotating key"
    return_body=""

    old_key_id=$(aws configure get aws_access_key_id --profile $realm)
    verbose "old key = ${old_key_id}"

    verbose "creating new access key"
    return_body=$(aws --profile=$realm iam create-access-key --user-name $username)
    verbose "return body ="
    verbose $return_body

    verbose "keys are:"
    new_key_id=$(echo $return_body | jq -r '.AccessKey | .AccessKeyId')
    verbose "new_key_id = ${new_key_id}"
    new_secret=$(echo $return_body | jq -r '.AccessKey | .SecretAccessKey')
    verbose "new_secret = ${new_secret}"

    verbose "deleting old access key"
    return_body=$(aws --profile=$realm iam delete-access-key --user-name $username --access-key-id $old_key_id)
    verbose "return body ="
    verbose $return_body

    verbose "setting aws_access_key_id"
    aws configure set profile.${realm}.aws_access_key_id $new_key_id
    verbose "setting aws_secret_access_key"
    aws configure set profile.${realm}.aws_secret_access_key $new_secret
  fi

  verbose ""
  verbose "SETTING MFA PROFILE"
  verbose "setting aws_access_key_id: aws configure set profile.${realm}_mfa.aws_access_key_id $access_key_id"
  aws configure set profile.${realm}_mfa.aws_access_key_id $access_key_id
  verbose "setting aws_secret_access_key: aws configure set profile.${realm}_mfa.aws_secret_access_key $secret_access_key"
  aws configure set profile.${realm}_mfa.aws_secret_access_key $secret_access_key
  verbose "setting aws_session_token: aws configure set profile.${realm}_mfa.aws_session_token $aws_session_token"
  aws configure set profile.${realm}_mfa.aws_session_token $aws_session_token
  verbose ""

  verbose "checking skip realm config is 0. it is = ${skip_realm_config}"
  if [[ $skip_realm_config -eq 0 ]]; then
    verbose "doing realm config"

    verbose "getting aws config for roles"
    return_body=$(aws configure get profile.${realm}.roles)
    verbose "return body ="
    verbose $return_body
    IFS=', ' read -r -a roles <<<"$return_body"

    for role in "${roles[@]}"; do
      verbose "role read: ${role}"
    done

    verbose "getting aws config for accounts"
    return_body=$(aws configure get profile.${realm}.accounts)
    verbose "return body ="
    verbose $return_body
    IFS=', ' read -r -a accounts <<<"$return_body"

    for account in "${accounts[@]}"; do
      verbose "getting account number from config for ${account}"
      account_number=$(aws configure get profile.${realm}.account_numbers.${account})
      verbose "account number is = ${account_number}"
      for role in "${roles[@]}"; do
        verbose "setting ${realm}_${account}_${role} source_profile = ${realm}_mfa"
        aws configure set profile.${realm}_${account}_${role}.source_profile ${realm}_mfa
        verbose "setting ${realm}_${account}_${role} role_arn = arn:aws:iam::${account_number}:role/${role}"
        aws configure set profile.${realm}_${account}_${role}.role_arn arn:aws:iam::${account_number}:role/${role}
      done
      if [[ $realm != "org_master" ]]; then
        verbose "linking account to org_master OrganizationAccountAccessRole profile"
        aws configure set profile.org_master_${realm}_${account}_OrganizationAccountAccessRole.source_profile org_master_mfa
        aws configure set profile.org_master_${realm}_${account}_OrganizationAccountAccessRole.role_arn arn:aws:iam::${account_number}:role/OrganizationAccountAccessRole
      fi
    done
  fi

  ####################################################
  ############### End Script Here ####################
}

############## Begin Options and Usage ###################

# Print usage
usage() {
  echo -n "${scriptName} [OPTION]... [FILE]...
This generates ~/.aws/credentials via the aws cli for mfa authentication.
username and account_numbers must be set in your realm's .aws/credentials profile.
Also, rotates your aws_access_key_id and secret key along with it each run unless you disable it.
Also, configures an entire realm based off of your ~/.aws/config and credentials. See README.md
 Options:
  -c, --code          required: Your rotating mfa code
  -r, --realm         required: The name of the realm. will result as realm_mfa as profile name
  -r, --region        change the region from default
  --skip-key-rotate   include this flag to skip the accesss key rotation
  --skip-realm-config include this flag to skip auto config of the entire realm in your ~/.aws/credentials file
  --duration-seconds  duration seconds the mfa is valid for. default is 129600 seconds(36 hr)
  -q, --quiet         Quiet (no output)
  -l, --log           Print log to file
  -s, --strict        Exit script with null variables.  i.e 'set -o nounset'
  -v, --verbose       Output more information. (Items echoed to 'verbose')
  -d, --debug         Runs script in BASH debug mode (set -x)
  -h, --help          Display this help and exit
      --version       Output version information and exit
"
}

# Iterate over options breaking -ab into -a -b when needed and --foo=bar into
# --foo bar
optstring=h
unset options
while (($#)); do
  case $1 in
  # If option is of type -ab
  -[!-]?*)
    # Loop over each character starting with the second
    for ((i = 1; i < ${#1}; i++)); do
      c=${1:i:1}

      # Add current char to options
      options+=("-$c")

      # If option takes a required argument, and it's not the last char make
      # the rest of the string its argument
      if [[ $optstring == *"$c:"* && ${1:i+1} ]]; then
        options+=("${1:i+1}")
        break
      fi
    done
    ;;

  # If option is of type --foo=bar
  --?*=*) options+=("${1%%=*}" "${1#*=}") ;;
  # add --endopts for --
  --) options+=(--endopts) ;;
  # Otherwise, nothing special
  *) options+=("$1") ;;
  esac
  shift
done
set -- "${options[@]}"
unset options

# Print help if no arguments were passed.
# Uncomment to force arguments when invoking the script
# [[ $# -eq 0 ]] && set -- "--help"

# Read the options and set stuff
while [[ $1 == -?* ]]; do
  case $1 in
  -c | --code)
    code=$2
    shift
    ;;
  -r | --realm)
    realm=$2
    shift
    ;;
  --region)
    region=$2
    shift
    ;;
  --mfa_arn)
    mfa_arn=$2
    shift
    ;;
  --duration-seconds)
    duration_seconds=$2
    shift
    ;;
  --skip-key-rotate) skip_key_rotate=1 ;;
  --skip-realm-config) skip_realm_config=1 ;;
  -h | --help)
    usage >&2
    exit 0
    ;;
  --version)
    echo "$(basename $0) ${version}"
    exit 0
    ;;
  -v | --verbose) verbose=1 ;;
  -l | --log) printLog=1 ;;
  -q | --quiet) quiet=1 ;;
  -s | --strict) strict=1 ;;
  -d | --debug) debug=1 ;;
  --force) force=1 ;;
  --endopts)
    shift
    break
    ;;
  *)
    echo "invalid option: '$1'."
    exit 1
    ;;
  esac
  shift
done

# Store the remaining part as arguments.
args+=("$@")

############## End Options and Usage ###################

# ############# ############# #############
# ##       TIME TO RUN THE SCRIPT        ##
# ##                                     ##
# ## You shouldn't need to edit anything ##
# ## beneath this line                   ##
# ##                                     ##
# ############# ############# #############

# Trap bad exits with your cleanup function
# trap trapCleanup EXIT INT TERM

# Exit on error. Append '||true' when you run the script if you expect an error.
set -o errexit

# Run in debug mode, if set
if [ "${debug}" == "1" ]; then
  set -x
fi

# Exit on empty variable
if [ "${strict}" == "1" ]; then
  set -o nounset
fi

# Bash will remember & return the highest exitcode in a chain of pipes.
# This way you can catch the error in case mysqldump fails in `mysqldump |gzip`, for example.
set -o pipefail

# Invoke the checkDependenices function to test for Bash packages
# checkDependencies

# Run your script
mainScript

# safeExit # Exit cleanly