How to create a folder in an amazon S3 bucket using terraform

Question

I was able to create a bucket in an amazon S3 using this link .

I used the following code to create a bucket :

resource "aws_s3_bucket" "b" {
    bucket = "my_tf_test_bucket"
    acl    = "private"
}

Now I wanted to create folders inside the bucket, say Folder1 .

I found the link for creating an S3 object. But this has a mandatory parameter source . I am not sure what this value have to , since my intent is to create a folder inside the S3 bucket.

Answer 1

For running terraform on Mac or Linux, the following will do what you want

resource "aws_s3_bucket_object" "folder1" {
    bucket = "${aws_s3_bucket.b.id}"
    acl    = "private"
    key    = "Folder1/"
    source = "/dev/null"
}

If you're on windows you can use an empty file.

While folks will be pedantic about s3 not having folders, there are a number of operations where having an object placeholder for a key prefix (otherwise called a folder) make life easier. Like s3 sync for example.

Answer 2

Actually, there is a canonical way to create it, without being OS dependent, by inspecting the Network on a UI put you see the content headers, as stated by : https://stackoverflow.com/users/1554386/alastair-mccormack ,

And S3 does support folders these days as visible from the UI.

So this is how you can achieve it:

resource "aws_s3_bucket_object" "base_folder" {
    bucket  = "${aws_s3_bucket.default.id}"
    acl     = "private"
    key     =  "${var.named_folder}/"
    content_type = "application/x-directory"
    kms_key_id = "key_arn_if_used"
}

Please notice the trailing slash otherwise it creates an empty file

Above has been used with a Windows OS to successfully create a folder using terraform s3_bucket_object.

Answer 3

S3 doesn't support folders. Objects can have prefix names with slashes that look like folders, but that's just part of the object name. So there's no way to create a folder in terraform or anything else, because there's no such thing as a folder in S3.

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html http://docs.aws.amazon.com/AWSImportExport/latest/DG/ManipulatingS3KeyNames.html

If you want to pretend, you could create a zero-byte object in the bucket named "Folder1/" but that's not required. You can just create objects with key names like "Folder1/File1" and it will work.

Answer 4

The answers here are outdated, it's now definitely possible to create an empty folder in S3 via Terraform. Using the aws_s3_object resource, as follows:

resource "aws_s3_bucket" "this_bucket" {
  bucket = "demo_bucket"
}

resource "aws_s3_object" "object" {
  bucket = aws_s3_bucket.this_bucket.id
  key    = "demo/directory/"
}

If you don't supply a source for the object then terraform will create an empty directory.

IMPORTANT - Note the trailing slash this will ensure you get a directory and not an empty file

Answer 5

old answer but if you specify the key with the folder (that doesn't exist yet) terraform will create the folder automatically for you

terraform {
  backend "s3" {
    bucket  = "mysql-staging"
    key     = "rds-mysql-state/terraform.tfstate"
    region  = "us-west-2"
    encrypt = true
  }
}

Answer 6

v0.12.8 introduces a new fileset() function which can be used in combination with for_each to support this natively :

NEW FEATURES:

lang/funcs: New fileset function, for finding static local files that match a glob pattern. ( #22523 )

A sample usage of this function is as follows (from here ):

# Given the file structure from the initial issue:
# my-dir
#    |- file_1
#    |- dir_a
#    |     |- file_a_1
#    |     |- file_a_2
#    |- dir_b
#    |     |- file_b_1
#    |- dir_c
# And given the expected behavior of the base_s3_key prefix in the initial issue

resource "aws_s3_bucket_object" "example" {
  for_each = fileset(path.module, "my-dir/**/file_*")

  bucket = aws_s3_bucket.example.id
  key    = replace(each.value, "my-dir", "base_s3_key")
  source = each.value
}

At the time of this writing, v0.12.8 is a day old (Released on 2019-09-04) so the documentation on https://www.terraform.io/docs/providers/aws/r/s3_bucket_object.html does not yet reference it. I am not certain if that's intentional.

---

As an aside, if you use the above, remember to update/create version.tf in your project like so:

terraform {
  required_version = ">= 0.12.8"
}

Answer 7

I would like to add to this discussion that you can create a set of empty folders by providing the resource a set of strings:

resource "aws_s3_object" "default_s3_content" {
    for_each = var.default_s3_content
    bucket = aws_s3_bucket.bucket.id
    key = "${each.value}/"
}

where var.default_s3_content is a set of strings:

variable "default_s3_content" {
   description = "The default content of the s3 bucket upon creation of the bucket"
   type = set(string)
   default = ["folder1", "folder2", "folder3", "folder4", "folder5"]
}