How can I create a custom Linux AMI with Packer that accepts user data

Question

The goal is to create an AMI that the end user can customize by including an external database or other instance specific configuration.

I have created a custom image that pulls down some assets from a repo as part of the Packer build, then creates a service which runs a script on next boot. When I launch the instance from this AMI, I include a user data script which just creates a prop file, then echoes some values into the prop file. Then the service script creates a prop file from user data if it exists or from default values, then starts the assets with those props.

However, after I start the instance, the file is not created by the user data. I've found some stuff saying this is a run once per instance id, but this is a new instance.

I've tried fetching the user data as part of the service script, wget http://169.254.169.254/latest/user-data , removing the state files associated with user data as part of the packer build process so the user_data will be run again when the AMI is instantiated, rm -Rf /var/lib/cloud/* , prepending the user data with #cloud-boothook so that it runs every time instead of just first boot. However, the tmp file from the user data script is not present, and the assets only start with default values.

Packer build.json

...
{
  "type": "file",
  "source": "../scripts/bootstrap.sh",
  "destination": "/tmp/bootstrap.sh"
},
{
  "type": "file",
  "source": "../scripts/myservice.service",
  "destination": "/tmp/myservice.service"
},
{
  "type": "shell",
  "environment_vars": [
    "REPO_USERNAME={{user `repo_user`}}",
    "REPO_PASSWORD={{user `rep_password`}}"
  ],
  "execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",
  "inline": [
    "cd /tmp",
    "chmod +x bootstrap.sh",
    "./bootstrap.sh"
  ]
}
...

bootstrap.sh

<pull assets>
...
mv /tmp/myservice.service /lib/systemd/system/myservice.service
sudo systemctl enable myservice.service
...

myservice.service

if [ -ne ./user_data_retrieved_flag ]
then
   wget http://169.254.169.254/latest/user-data
   touch user_data_retrieved_flag
fi
if [ -e ./user-data ] && [ -ne ./user_data_processed_flag ]
then
  chmod 755 ./user-data
  ./user-data
  touch user_data_processed_flag
fi
if [ -e /tmp/my_props.properties ]
then
    cat /tmp/my_props
    mv /tmp/my_props /home/ubuntu/my_props
fi
<myResources.start>
...

Terraform

...
resource "aws_instance" "my_instance" {
  ami = "${var.my_custom_ami}"
  ...
  user_data = <<EOF
    #cloud-boothook
    #!/bin/bash
    touch /tmp/my_props
    echo PROP1=${var.prop1} >> /tmp/my_props
    echo PROP2_USER=${var.db_un} >> /tmp/my_props
    chown ubuntu:ubuntu /tmp/my_props
    chmod 777 /tmp/my_props
    EOF
}
...

Edit: To clarify, the assets do run, so I know the service is executing successfully. However, they are using default props and neither the flags mentioned in configure.sh, or the file which should have been created by the user data are present.

Answer 1

User data scripts are executed via cloud-init , a daemon that can configure instances at creation/boot time independently from what is baked in to the image.

As such, if you want to use user data you probably want to make sure that cloud-init is installed in your image. The easiest option here is to simple create your AMI from a pre-existing AMI that already has cloud-init installed such as Amazon Linux, the official Canonical provided Ubuntu AMIs, the official Red Hat provided Red Hat images etc. Alternatively you should be able to install it via your distro's package manager.

If you want a minimal way of executing user data scripts in AWS without cloud-init (such as for distros where cloud-init isn't available such as OpenBSD) you could use something like this:

#!/bin/sh

# Ghetto cloud-init to execute EC2 user data scripts

ID="$(curl --silent 169.254.169.254/latest/meta-data/instance-id)"

if [ ! -f /var/lib/cloud/instance/boot-finished ] && [ "$(cat /var/lib/cloud/instance/boot-finished)" != "${ID}" ]; then
  curl --silent 169.254.169.254/latest/user-data -o /tmp/user-data
  chmod +x /tmp/user-data

  /tmp/user-data >> /var/log/cloud-init-output.log 2>&1

  mkdir -p /var/lib/cloud/instance/
  echo "${ID}" > /var/lib/cloud/instance/boot-finished
fi

And run this as a daemon, making sure it starts very early in the boot process.

This will scrape the user data script from the EC2 metadata endpoint and then execute it, writing the output to the typical log location for cloud-init and then make sure it is only executed on first boot with a semaphore file that it checks for before running.