Create EBS snapshot using Terraform and count()

Question

I want to create snapshots for all gp2 type EBS volumes. I have the following code:

data "aws_ebs_volume" "ebs_volumes" {
  filter {
    name   = "volume-type"
    values = ["gp2"]
  }
}

resource "aws_ebs_snapshot" "ebs_snapshot" {
  count = "${length(data.aws_ebs_volume.ebs_volumes.ids)}"
  volume_id = "${element(data.ebs_volume.ebs_volumes.ids, count.index)}"
}

Instead I got the following error:

terraform plan

Error: resource 'aws_ebs_snapshot.ebs_snapshot' config: unknown resource 'data.ebs_volume.ebs_volumes' referenced in variable data.ebs_volume.ebs_volumes.ids

Since I am using data source, the list returns by data.aws_ebs_volume is dynamic - not a static variable as discussed in Create snapshots of multiple EBS volumes using Terraform .

Using terraform show :

data.aws_ebs_volume.ebs_volumes:
  id = vol-00b3eaaf04b9377cb
  arn = arn:aws:ec2:us-east-1:ACCOUNT_ID:volume/vol-00b3eaaf04b9377cb
  availability_zone = us-east-1c
  encrypted = false
  filter.# = 1
  filter.3737401200.name = volume-type
  filter.3737401200.values.# = 1
  filter.3737401200.values.0 = gp2
  iops = 100
  kms_key_id =
  most_recent = false
  size = 8
  snapshot_id = snap-01d81204beb02804b
  tags.% = 0
  volume_id = vol-00b3eaaf04b9377cb
  volume_type = gp2

Answer 1

There are normally 2 types of data source in Terraform providers, singular ones and plural ones such as aws_ami vs aws_ami_ids with the plural one normally just returning a list of IDs of resources and the singular one offering more information about each specific resource.

Unfortunately the AWS provider has not yet implemented a plural data source for EBS volumes so you are limited to just the singular aws_ebs_volume data source and can't just dynamically return all EBS volumes matching some criteria (such as them being GP2).

As a short term fix, and if you really want to use Terraform to manage creating snapshots directly, you could use an external data source to get the list of EBS volumes outside of Terraform.

A quick example of this might look something like this (untested):

data "external" "all_gp2_ebs_volumes" {
  # Should run something like:
  # `aws ec2 describe-volumes --query 'Volumes[].VolumeId' --filters Name=volume-type,Values=gp2`
  # but return the result in a way that the data source needs it.
  program = ["python", "${path.module}/get-all-gp2-ebs-volumes.py"]
}

resource "aws_ebs_snapshot" "ebs_snapshot" {
  count     = "${length(data.external.all_gp2_ebs_volumes.result.ids)}
  volume_id = "${data.external.all_gp2_ebs_volumes.result.ids[count.index]}"
}

Unless the 2 hour period restriction on AWS DLM or AWS Backup is a real blocker for you for some reason I would use those. There is already an aws_dlm_lifecycle_policy resource to configure this via Terraform and then this issue is tracking the creation of the necessary AWS Backup resources.

Another alternative would be to raise a feature request on the issue tracker for a aws_ebs_volume_ids data source (I couldn't find an existing feature request from a quick look).