Is there a way to append IAM Bucket Policy Statement to Existing Statements using terraform?

Question

I currently have a centralised S3 bucket in a central account accessed via multiple other child accounts. I would like to dynamically update the bucket policy from each of the child accounts by taking the existing IAM Bucket Policy and append a new statement allowing the IAM role from the calling account to have access to the bucket.

I tried to use a combination of aws_iam_policy_document and source_json with a new statement. However, all this does is overwrite the existing statement.

What I am trying to achieve is take this existing policy json and either merge or append to a new statement. Any ideas how to achieve this?

Does anyone have a working example of managing cross account s3 bucket policies and dynamically updating the policy to allow access from roles within child accounts?

Answer 1

The aws_iam_policy_document data source's source_json argument works by merging policy statements using their statement id ("sid") values, so in order for statements from the previous JSON to appear in the result the sid arguments in the new statements must be distinct from the ones in the source document.

Another option is to do the transformations more manually within the Terraform language itself. This requires working directly with the raw statement data structures, and so it'll be your responsibility to ensure that you handle the input robustly and produce a valid IAM policy data structure as the result.

For example:

locals {
  policy_a = jsondecode(file("${path.module}/policy_a.json"))
  policy_b = jsondecode(file("${path.module}/policy_b.json"))
  policy_c = {
    Version: local.policy_a.Version,
    Statement: concat(
      local.policy_a.Statement,
      local.policy_b.Statement,
    ),
  }
}

You can then produce a JSON version of policy_c somewhere else in your module using jsonencode(local.policy_c) .

Because this is using concat , the result will be literally the two lists of statements concatenated together, so you'll need to ensure that the result is sensible yourself: there will be no automatic overriding by statement id or any similar normalizations.

Answer 2

I had a similar situation and my workaround was complicated, involving external scripts and a null resource. In my situation, I was using multiple terraform workspaces, each with their own cloudfront distribution, but all of them used the S3 bucket for one of the origins. For each workspace, I needed to append the workspace's cloudfront origin access identity to the bucket's policy, without replacing or destroying the entries that already existed in that policy.

The bucket was created manually (outside of terraform) and imported it as a data resource.

I had 3 external scripts (though it could be a single more complicated script) - one to create a policy with the new identity, one to create a policy WITHOUT the identity (used for destroy ), and one to actually apply the created policies (used by the null resource, calling the aws command directly).

resource "aws_cloudfront_origin_access_identity" "s3_identity" {
  comment = terraform.workspace
}

data "external" "append_bucket_policy" {
  program = ["scripts/append_bucket_policy.sh", data.aws_s3_bucket.bucket.id, aws_cloudfront_origin_access_identity.s3_identity.iam_arn]
}                                                   
     
data "external" "cleanup_bucket_policy" {
  program = ["scripts/cleanup_bucket_policy.sh", data.aws_s3_bucket.bucket.id, aws_cloudfront_origin_access_identity.s3_identity.id]
}

locals {
  bucket_identifiers      = compact(split(",", data.external.append_bucket_policy.result["principals"]))
  cleanup_policy_identifiers = compact(split(",", data.external.cleanup_bucket_policy.result["principals"]))
}

data "aws_s3_bucket" "bucket" {
  bucket   = "bucketname"       
}

data "aws_iam_policy_document" "append_policy" {
  statement {
    actions   = ["s3:GetObject"]
    resources = ["${data.aws_s3_bucket.bucket.arn}/*"]

    principals {
      type        = "AWS"
      identifiers = local.bucket_identifiers
    }
  }

  statement {
    actions   = ["s3:ListBucket"]
    resources = [data.aws_s3_bucket.bucket.arn]

    principals {
      type        = "AWS"
      identifiers = local.bucket_identifiers
    }
  }
}


data "aws_iam_policy_document" "cleanup_policy" {
  statement {
    actions   = ["s3:GetObject"]
    resources = ["${data.aws_s3_bucket.bucket.arn}/*"]

    principals {
      type        = "AWS"
      identifiers = local.cleanup_policy_identifiers
    }
  }

  statement {
    actions   = ["s3:ListBucket"]
    resources = [data.aws_s3_bucket.bucket.arn]

    principals {
      type        = "AWS"
      identifiers = local.cleanup_policy_identifiers
    }
  }
}

resource "null_resource" "apply_policy" {
  depends_on = [data.external.append_bucket_policy, data.external.append_bucket_policy, data.aws_iam_policy_document.append_policy, data.external.cleanup_bucket_policy, data.aws_iam_policy_document.cleanup_policy, aws_cloudfront_origin_access_identity.s3_identity]

  provisioner "local-exec" {
    when    = create
    command = "scripts/apply_bucket_policy.sh ${data.aws_s3_bucket.bucket.id} '${data.aws_iam_policy_document.append_policy.json}'"
  }

  provisioner "local-exec" {
    when    = destroy
    command = "scripts/apply_bucket_policy.sh ${data.aws_s3_bucket.bucket.id} '${data.aws_iam_policy_document.cleanup_policy.json}'"
  }
}

append_bucket_policy.sh

#!/bin/bash

set -eo pipefail

# To be called as an external data resource program by terraform
# Usage: append_bucket_policy.sh <bucket name> <principal to append>
#
# Retrieves the current principals in bucket policy for <bucket name>, appends <principal to append>, uniqs the output to remove any duplicates, concats the principals into a comma seperated string, and outputs a json blob that terraform can understand
# In terraform, the variable is split on commas to create a list, which is used in the "identifiers" of an iam policy document resource, and re-applied to the bucket

IFS='
'

# this likely won't work with more complex bucket policies, but it does the job for cloudfront origin access identities
CURRENT=$(aws --region us-west-2 s3api get-bucket-policy --bucket "${1}" | jq --raw-output '.Policy' | jq '.Statement[].Principal.AWS[]')

TEMP=$(for i in ${CURRENT} ; do
  printf "%s\n" "${i},"
done
printf "\"%s\"" "${2},")

NEW=$(echo "${TEMP}" | sort | uniq | tr -d '\n')

jq -n --arg principals "${NEW}" '.principals = $principals' | sed 's/\\\"//g'

cleanup_bucket_policy.sh

#!/bin/bash

set -eo pipefail

IFS='
'

CURRENT=$(aws --region us-west-2 s3api get-bucket-policy --bucket "${1}" | jq --raw-output '.Policy' | jq '.Statement[].Principal.AWS[]')

TEMP=$(for i in ${CURRENT} ; do
  printf "%s\n" "${i},"
done | egrep -v "${2}")

NEW=$(echo "${TEMP}" | sort | uniq | tr -d '\n')

jq -n --arg principals "${NEW}" '.principals = $principals' | sed 's/\\\"//g'

apply_bucket_policy.sh

#!/bin/bash

# apply bucket policy generated by append_bucket_policy.sh

set -eo pipefail

# may fail to apply if the origin access identity was just created, wait a few seconds
sleep 10

RESULT=$(aws --region us-west-2 s3api put-bucket-policy --bucket "${1}" --policy "${2}")

jq -n --arg result "${RESULT}" '.result = $result' | sed 's/\\\"//g'

I apologize for this being terribly complicated and maybe not a 100% fit, but it worked for me in the past and might spark some ideas for you. I welcome a better way to do this, too!