如何使用 terraform 将生命周期规则添加到 S3 存储桶？

Question

I am using Terraform to create a bucket in S3 and I want to add "folders" and lifecycle rules to it.

I can create the bucket (using an "aws_s3_bucket" resource).

I can create the bucket and define my lifecycle rules within the same "aws_s3_bucket" resource, ie. at creation time.

I can add "folders" to the bucket (I know they aren't really folders, but they are presented to the client systems as if they were... :-) ), using an "aws_s3_bucket_object" resource, ie. after bucket creation.

All good...

But I want to be able to add lifecycle rules AFTER I've created the bucket, but I get an error telling me the bucket already exists. (Actually I want to be able to subsequently add folders and corresponding lifecycle rules as and when required.)

Now, I can add lifecycle rules to an existing bucket in the AWS GUI, so I know it is a reasonable thing to want to do.

But is there a way of doing it with Terraform?

Am I missing something?

resource "aws_s3_bucket" "bucket" {
    bucket      = "${replace(var.tags["Name"],"/_/","-")}"
    region      = "${var.aws_region}"

    #tags                = "${merge(var.tags, map("Name", "${var.tags["Name"]}"))}"
    tags                = "${merge(var.tags, map("Name", "${replace(var.tags["Name"],"/_/","-")}"))}"
}


resource "aws_s3_bucket" "bucket_quarterly" {
    bucket      = "${aws_s3_bucket.bucket.id}"
    #region      = "${var.aws_region}"

    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "quarterly/"
        enabled = true

        expiration {
            days = 92
        }
    }

}


resource "aws_s3_bucket" "bucket_permanent" {
    bucket      = "${aws_s3_bucket.bucket.id}"
    #region      = "${var.aws_region}"

    lifecycle_rule {
        id      = "permanent_retention"
        enabled = true
        prefix  = "permanent/"

        transition {
            days            = 1
            storage_class   = "GLACIER"
        }
    }

}


resource "aws_s3_bucket_object" "quarterly" {
    bucket  = "${aws_s3_bucket.bucket.id}"
    #bucket  = "${var.bucket_id}"
    acl     = "private"
    key     = "quarterly"
    source  = "/dev/null"
}


resource "aws_s3_bucket_object" "permanent" {
    bucket  = "${aws_s3_bucket.bucket.id}"
    #bucket  = "${var.bucket_id}"
    acl     = "private"
    key     = "permanent"
    source  = "/dev/null"
}

I expect to have a bucket with 2 lifecycle rules, but I get the following error:

Error: Error applying plan:

2 error(s) occurred:

* module.s3.aws_s3_bucket.bucket_quarterly: 1 error(s) occurred:

* aws_s3_bucket.bucket_quarterly: Error creating S3 bucket: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
    status code: 409, request id: EFE9C62B25341478, host id: hcsCNracNrpTJZ4QdU0AV2wNm/FqhYSEY4KieQ+zSHNsj6AUR69XvPF+0BiW4ZOpfgIoqwFoXkI=
* module.s3.aws_s3_bucket.bucket_permanent: 1 error(s) occurred:

* aws_s3_bucket.bucket_permanent: Error creating S3 bucket: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
    status code: 409, request id: 7DE1B1A36138A614, host id: 8jB6l7d6Hc6CZFgQSLQRMJg4wtvnrSL6Yp5R4RScq+GtuMW+6rkN39bcTUwQhzxeI7jRStgLXSc=

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Answer 1

Lets first break down whats happening and how we can overcome this issue. Each time you define a resource "aws_s3_bucket" , terraform will attempt to create a bucket with the parameters specified. If you want to attach a lifecycle policy to a bucket, do it where you define the bucket, eg:

resource "aws_s3_bucket" "quarterly" {
    bucket  = "quarterly_bucket_name"
    #bucket  = "${var.bucket_id}"
    acl     = "private"
    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "folder/"
        enabled = true

        expiration {
            days = 92
        }
    }
}

resource "aws_s3_bucket" "permanent" {
    bucket  = "perm_bucket_name"
    acl     = "private"
    lifecycle_rule {
        id      = "permanent_retention"
        enabled = true
        prefix  = "permanent/"

        transition {
            days            = 1
            storage_class   = "GLACIER"
        }
    }
}

A bucket can have multiple lifecycle_rule blocks on it. If you want to define the lifecycle rules as external blocks, you can do it in this way:

// example of what the variable would look like:
variable "lifecycle_rules" {
  type = "list"
  default = []
}

// example of what the assignment would look like:
lifecycle_rules = [{
  id = "cleanup"
  prefix = ""
  enabled = true
  expiration = [{
    days = 1
  }]
}, {...}, {...} etc...]

// example what the usage would look like
resource "aws_s3_bucket" "quarterly" {
    bucket  = "quarterly_bucket_name"
    #bucket  = "${var.bucket_id}"
    acl     = "private"
    source  = "/dev/null"
    lifecycle_rule = [ "${var.lifecycle_rules}" ]
}

Note: the implementation above of having an external lifecycle policy isn't really the best way to do it, but the only way. You pretty much trick terraform into accepting the list of maps, which happens to be the same type as lifecycle_rule, so it works. Ideally, Terraform should have it's own resource block for lifecycle rules, but it doesn't.

Edit: why have separate resource blocks when we now have dynamic blocks! Woohoo

Answer 2

As far as I am aware, you cannot make a lifecycle policy separately.

Someone raised a PR for a resource to be created to allow you to do so, but looks like it is still open: https://github.com/terraform-providers/terraform-provider-aws/issues/6188

---

As for your error, I believe the reason you're getting the error is because:

• resource "aws_s3_bucket" "bucket"

Creates a bucket with a particular name.

• resource "aws_s3_bucket" "bucket_quarterly"

References bucket = "${aws_s3_bucket.bucket.id}" and therefore tries to create a bucket with the same name as the previous resource (which cannot be done as names are unique).

• resource "aws_s3_bucket" "bucket_permanent"

Similarly, this resource references bucket = "${aws_s3_bucket.bucket.id}" and therefore tries to create a bucket with the same name as the first resource (which cannot be done as names are unique).

---

You mentioned I expect to have a bucket with 2 lifecycle rules but in your above code you are creating 3 separate s3 buckets (one without a lifecycle, and 2 with a lifecycle) and two objects (folders) that are being placed into the s3 bucket without a lifecycle policy.

Answer 3

Thanks for the info (I like the idea of the list to separate the rules from the resource).

The issue was that I didn't appreciate that you could define lifecycle rules within the resource AND change them subsequently, so I was trying to figure out how to define them separately...

All that's required is to specify them in the resource and do terraform apply, then you can edit it and add/amend/remove lifecycle_rules items and just do terraform apply again to apply the changes.

Answer 4

    source "aws_s3_bucket" "my_s3_bucket" {
       bucket = local.s3_bucket_name
    
    }



    
    resource "aws_s3_bucket_acl" "my_s3_bucket_acl" {
      bucket = aws_s3_bucket.my_s3_bucket.arn
    
      lifecycle {
        prevent_destroy = true
      }
    }


    
    resource "aws_s3_bucket_versioning" "my_s3_bucket_versioning" {
      bucket = aws_s3_bucket.my_s3_bucket.arn
      versioning_configuration {
        status = true
      }
    }
    



    resource "aws_s3_bucket_server_side_encryption_configuration" "my_s3-bucket_encryption" {
      bucket = aws_s3_bucket.my_s3_bucket.arn
      rule {
        apply_server_side_encryption_by_default {
          sse_algorithm = "AES256"
        }
      }

}

resource "aws_s3_bucket_lifecycle_configuration" "my_s3_bucket_lifecycle_config" {
  bucket = aws_s3_bucket.my_s3_bucket.arn
  rule {
    id     = "dev_lifecycle_7_days"
    status = true

    abort_incomplete_multipart_upload {
      days_after_initiation = 30
    }
    noncurrent_version_expiration {
      noncurrent_days = 1
    }
    transition {
      storage_class = "STANDARD_IA"
      days = 30
    }
    expiration {
      days = 30
    }
  }
}