How can I chain packer output ami id to terraform variables automatically?

Question

I'm using packer with ansible provisioner to build an ami, and terraform to setup the infrastructure with that ami as a source - somewhat similar to this article: http://www.paulstack.co.uk/blog/2016/01/02/building-an-elasticsearch-cluster-in-aws-with-packer-and-terraform

When command packer build pack.json completes successfully I get the output ami id in this format:

eu-central-1: ami-12345678

In my terraform variables variables.tf I need to specify the source ami id, region etc. The problem here is that I don't want to specify them manually or multiple times. For region (that I know beforehand) it's easy since I can use environment variables in both situations, but what about the output ami? Is there a built-in way to chain these products or some not so hacky approach to do it?

EDIT: Hacky approach for anyone who might be interested. In this solution I'm grep ing the aws region & ami from packer output and use a regular expression in perl to write the result into a terraform.tfvars file:

vars=$(pwd)"/terraform.tfvars"
packer build pack.json | \
    tee /dev/tty | \
    grep -E -o '\w{2}-\w+-\w{1}: ami-\w+' | \
    perl -ne '@parts = split /[:,\s]+/, $_; print "aws_amis." . $parts[0] ." = \"" . $parts[1] . "\"\n"' > ${vars}

Answer 1

You should consider using Terraform's Data Source for aws_ami . With this, you can rely on custom tags that you set on the AMI when it is created (for example a version number or timestamp). Then, in the Terraform configuration, you can simply filter the available AMIs for this account and region to get the AMI ID that you need.

https://www.terraform.io/docs/providers/aws/d/ami.html

data "aws_ami" "nat_ami" {
  most_recent = true
  executable_users = ["self"]
  filter {
    name = "owner-alias"
    values = ["amazon"]
  }
  filter {
    name = "name"
    values = ["amzn-ami-vpc-nat*"]
  }
  name_regex = "^myami-\\d{3}"
  owners = ["self"]
}

NOTE: in the example above (from the docs), the combination of filters is probably excessive. You can probably get by just fine with something like:

data "aws_ami" "image" {
  most_recent = true
  owners = ["self"]
  filter {                       
    name = "tag:Application"     
    values = ["my-app-name"]
  }                              
}

output "ami_id" {
  value = "${data.aws_ami.image.id}"
}

An additional benefit of this is that you can deploy to multiple regions with the same configuration and no variable map!

Answer 2

The "official" way that is recommended by Hashicorp is to use their product Atlas as a "middleman" between the two. You'd use the Atlas post-processor in Packer to record the artifacts (AMI ids, in your case) and then use the atlas_artifact resource in Terraform to read the ids back out again for use in Terraform.

In this case, you would obtain the ids from the resource rather than passing them in using variables.

Aside from Atlas the other options are rather limited, and in some cases hacky.

If you want to do it without any external services at all then you can experiment with the local shell post-processor as a way to run a local command on your artifact, or you can use the machine-readable output to extract the AMI ids and write them into a variables file for Terraform.

A further option is to write your own post-processor plugin that interacts with some software you already use, as an alternative to Atlas. For example, with some of my colleagues I wrote a post-processor to record artifacts as metadata in Buildkite , which we then subsequently retrieve using the Buildkite API . This requires writing custom code in Go.

At the time of writing Terraform version 0.7 is still under development, but it is planned to include a new feature that allows querying the EC2 API for AMIs directly, which will (if it indeed lands for 0.7) allow a further option of tagging the AMI with Packer and then finding it directly from EC2 using those tags. This uses EC2 itself as the "middleman", which is perhaps less awkward since it was involved already as the storage for the AMI anyway.

Answer 3

This is the approach that I used:

1. Wrap the packer call and get the AMI by parsing the output
2. Use the parsed AMI to create a Terraform files which provides the value as a variable

It is similar to the version in the edited answer. In more detail, it can look like this:

First, create a file called ami.tf.template :

# "ami.tf" was automatically generated from the template "ami.tf.template".
variable "ami" {
  default     = "${AMI_GENERATED_BY_PACKER}"
  description = "The latest AMI."
}

This template will be used to create the ami.tf file, which makes the AMI from packer available to your existing Terraform setup.

Second, create a shell wrapper script for running packer. You can use the following ideas:

# run packer (prints to stdout, but stores the output in a variable)
packer_out=$(packer build packer.json | tee /dev/tty)

# packer prints the id of the generated AMI in its last line
ami=$(echo "$packer_out" | tail -c 30 | perl -n -e'/: (ami-.+)$/ && print $1')

# create the 'ami.tf' file from the template:
export AMI_GENERATED_BY_PACKER="$ami" && envsubst < ami.tf.template > ami.tf

Once the script is done, it has created an ami.tf file, which may look like this:

# "ami.tf" was automatically generated from the template "ami.tf.template".
variable "ami" {
  default     = "ami-aa92a441"
  description = "The latest AMI."
}

Finally, put that file next to your existing Terraform setup. Then you can then access the AMI like this:

resource "aws_launch_configuration" "foo" {
  image_id = "${var.ami}"
  ...
}

Answer 4

I used the simplest approach, I gave my AMI a name, and looked up for the name in Terraform. image.pkr.hcl file -

packer {
  required_plugins {
    amazon = {
      version = ">= 0.0.2"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "ubuntu" {
  ami_name      = "your_ami_name"
  instance_type = "t2.micro"
  region        = "${var.aws_region}"
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = "${var.image_owners}"
  }
  ssh_username = "ubuntu"
}

build {
  name = "build-image"
  sources = [
    "source.amazon-ebs.ubuntu"
  ] 
#.....
}

and ec2.tf file is -

data "aws_ami" "ami" {
  most_recent = true
  owners = ["self"]
  filter {
    name = "name"
    values = ["your_ami_name"]
  }
}

output "ami_id" {
  value = data.aws_ami.ami.id
}

resource "aws_instance" "myEc2" {
  ami = data.aws_ami.ami.id
  instance_type = "t2.micro"
  key_name      = "----key"
  vpc_security_group_ids = [
    "launch-wizard-18"
  ]
    user_data = <<-EOL
      #!/bin/bash -xe
      echo "if done file exists it went ok"
      touch done.txt

      EOL

  tags = {
    Name = "child-of-your-ami"
  }
}

Hope this helps