在 VMware Ubuntu 22.04 客户机上通过 Packer 和 Terraform 运行 cloud-init 两次

Question

我想从 ESXi 中的同一个 Ubuntu 模板创建具有不同 CPU、RAM 和网络配置的虚拟机。

当$ packer build -var-file=packer/variables.pkr.hcl -var-file=packer/secret.pkrvars.hcl packer/template.pkr.hcl运行时，它会读取以下packer/template.pkr.hcl ：

variable "vm_name" {
  type = string
  default = "Ubuntu_Server_22.04_LTS"
}

variable "esxi_password" {
  type =  string
  default = "password"
  sensitive = true
}

variable "vm_password" {
  type =  string
  default = "password"
  sensitive = true
}

source "vmware-iso" "ubuntu-2204" {
  vm_name = "${var.vm_name}"
  guest_os_type = "ubuntu-64"

  iso_checksum = "sha256:84aeaf7823c8c61baa0ae862d0a06b03409394800000b3235854a6b38eb4856f"
  iso_url = "https://REDACTED/ubuntu-22.04-live-server-amd64.iso"

  http_directory = "/home/REDACTED/packer/http"

  shutdown_command = "sudo shutdown -P now"

  remote_type = "esx5"

  remote_datastore = "REDACTED"
  remote_host = "REDACTED"
  remote_username = "REDACTED"
  remote_password = "${var.esxi_password}"
  remote_private_key_file = ""

  cpus = 8
  memory = 16384
  disk_size = 16384

  network_adapter_type = "vmxnet3"
  network_name = "REDACTED"

  headless = false
  vnc_over_websocket = true
  insecure_connection = true

  tools_upload_flavor = "linux"

  skip_export = true
  keep_registered = true

  ssh_username = "REDACTED"
  ssh_password = "${var.vm_password}"
  ssh_timeout = "15m"
  ssh_handshake_attempts = "100"

  boot_wait = "3s"
  boot_command = [
    "<esc><esc><esc><esc>e<wait>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del>",
    "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://[{{.HTTPIP}}]:{{.HTTPPort}}/\"<enter><wait>",
    "initrd /casper/initrd<enter><wait>",
    "boot<enter>",
    "<enter><f10><wait>"
  ]
}

build {
  sources = ["sources.vmware-iso.ubuntu-2204"]
  provisioner "shell" {
    inline = [
      "ls /"
    ]
  }
}

packer/http/user-data包含以下内容：

#cloud-config
autoinstall:
  version: 1
  early-commands:
    # Stop SSH to prevent Packer from connecting too early
    - systemctl stop ssh
  apt:
    preserve_sources_list: false
    primary:
    - arches: [amd64, i386]
      uri: https://REDACTED
    - arches: [default]
      uri: http://ports.ubuntu.com/ubuntu-ports
  locale: en_US
  keyboard:
    layout: en
    variant: us
  network:
    version: 2
    renderer: networkd
    ethernets:
      ens160:
        dhcp4: true
        dhcp-identifier: mac
        dhcp6: true
  storage:
    layout:
      name: direct
    config:
      - type: disk
        id: disk0
        match:
          size: largest
      - type: partition
        id: boot-partition
        device: disk0
        size: 500M
      - type: partition
        id: root-partition
        device: disk0
        size: -1
  ssh:
    install-server: true
    allow-pw: true
    authorized-keys:
      - ssh-ed25519 REDACTED
  identity:
    hostname: ubuntu
    username: REDACTED
    password: REDACTED
  packages:
    - open-vm-tools
    - python3
  late-commands:
    - echo 'REDACTED ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/REDACTED
    - curtin in-target --target=/target -- chmod 440 /etc/sudoers.d/REDACTED
    - curtin in-target --target=/target -- apt-get update
    - curtin in-target --target=/target -- apt-get upgrade --yes
    - curtin in-target --target=/target -- sudo cloud-init clean

这将创建一个 Ubuntu 22.04 服务器模板，然后我可以使用 Terraform 从中配置虚拟机。

# cat /var/log/installer/autoinstall-user-data在 VM 上运行表明 Packer 已成功提供用户数据。 它已被执行，我通过 SSH 登录的能力证明了这一点。

当$ terraform apply -var-file=secret.tfvars在我的 terraform 目录中运行时，它会读取以下main.tf和https://github.com/josenk/terraform-provider-esxi提供的提供provider "esxi" ：

variable "vm_name" {
  description = "The name of the virtual machine"
  default     = "ubuntu-terraformed"
  type        = string
}

variable "esxi_password" {
  description = "The password for the ESXi root user"
  type        = string
}

provider "esxi" {
  esxi_hostname = "REDACTED"
  esxi_username = "REDACTED"
  esxi_password = "${var.esxi_password}"
}

data "template_file" "Test" {
  template = file("userdata.tpl")
  vars = {
    HOSTNAME = "${var.vm_name}"
  }
}

resource "esxi_guest" "Test" {
  guest_name = "${var.vm_name}"
  disk_store = "REDACTED"

  clone_from_vm = "Ubuntu_Server_22.04_LTS"

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  guestinfo = {
    "userdata.encoding" = "gzip+base64"
    "userdata"          = base64gzip(data.template_file.Test.rendered)
  }
}

userdata.tpl包含以下内容：

#cloud-config

hostname: ${HOSTNAME}
network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens161:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens192:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens193:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens224:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens225:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens256:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
package_upgrade: true
#ntp:
#  enabled: true
#  servers:
#    - REDACTED
#timezone: REDACTED
#late-commands:
#  - curtin in-target --target=/target -- sudo sed -i 's/#NTP=/NTP=REDACTED/g' /etc/systemd/timesyncd.conf
#  - curtin in-target --target=/target -- sudo timedatectl set-ntp true
#  - curtin in-target --target=/target -- sudo timedatectl set-timezone REDACTED
#  - curtin in-target --target=/target -- sudo systemctl restart systemd-timesyncd.service

这将基于带有正确来宾参数的打包程序模板创建一个 VM。 VMware 来宾配置包含我已验证的 userdata 属性与 base64 解码和解压缩来宾参数后提供的用户数据相匹配。

我遇到的问题是 VM 似乎不包含或执行“第二个”Terraform cloud-init 用户数据。

/var/lib/cloud/instance/user-data.txt 没有显示第二个配置：

#cloud-config
growpart:
  mode: 'off'
locale: en_US.UTF-8
preserve_hostname: true
resize_rootfs: false
ssh_pwauth: true
users:
- gecos: me
  groups: !!set
    adm: null
    cdrom: null
    dip: null
    lxd: null
    plugdev: null
    sudo: null
  lock_passwd: false
  name: me
  passwd: REDACTED
  shell: /bin/bash
  ssh_authorized_keys:
  - ssh-ed25519 REDACTED
    me

/var/log/cloud-init-output.log 显示：

schema.py[WARNING]: Invalid cloud-config provided:
users.0: {'gecos': 'me', 'groups': {'sudo', 'lxd', 'cdrom', 'adm', 'plugdev', 'dip'}, 'lock_passwd': False, 'name': 'me', 'passwd': 'REDACTED', 'shell': '/bin/bash', 'ssh_authorized_keys': ['ssh-ed25519 REDACTED me']} is not valid under any of the given schemas

当我的工作流程是 Packer 生成模板时，尝试应用用户数据两次，而 Terraform 使用该模板应用新 VM，对吗？ 如果是这样，我最好在哪里找出为什么 Ubuntu 不包含并执行第二次迭代？

顺便说一句，如果我有任何其他建议我应该考虑，请随时发表评论。 我想我应该能够通过即将实施的 Ansible 计划来实现这一点，但我仍然需要设置诸如主机名之类的选项。

Answer 1

我能够通过将 Packer 构建更改为以下内容来解决此问题：

build {
  sources = ["sources.vmware-iso.ubuntu-2204"]
  provisioner "shell" {
    inline = [
      "sudo rm -f /etc/cloud/cloud.cfg.d/99-installer.cfg",
      "sudo rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg",
      "echo 'disable_vmware_customization: false' | sudo tee -a /etc/cloud/cloud.cfg",
      "sudo sed -i 's|nocloud-net;seedfrom=http://.*/|vmware|' /etc/default/grub",
      "sudo update-grub",
      "sudo cloud-init clean"
    ]
  }
}

以下文章中引用了 cloud.cfg 调整，并允许通过 cloud-init 配置操作系统网络，这是让这一切正常工作的最后一步：

调整 GRUB 引导命令，并将cloud-init clean附加到 Packer 构建配置步骤可确保在 Packer 构建的客户机最终关闭之前，VM 将不再尝试从初始预置启动，而是将使用下次启动时指定的 VM guestinfo。

我还将 Terraform 用户数据拆分为用于实例标识和网络配置的元数据，将现在未注释的 NTP 和时区配置留在用户数据中。

元数据.tpl：

#cloud-config

instance-id: ${HOSTNAME}
local-hostname: ${HOSTNAME}
network:
  version: 2
  ethernets:
    nics:
      match:
        name: ens*
      dhcp-identifier: mac
      dhcp4: yes
      dhcp6: yes

用户数据.tpl：

#cloud-config

package_upgrade: true
ntp:
  enabled: true
  servers:
    - REDACTED
timezone: REDACTED

顺便说一句，配置网络接口的更简单方法是使用 match 指令而不是显式指定接口。

所有项目现在都按预期配置。 可能仅在使用vmware-iso的 ESXi 中才需要此配置，因为我没有看到其他地方需要这些步骤。

gecos 警告似乎是一个红鲱鱼，我没有进一步尝试解决这个问题。

在 VMware Ubuntu 22.04 客户机上通过 Packer 和 Terraform 运行 cloud-init 两次

问题描述

1 个解决方案

解决方案1
0 已采纳 2022-06-27 07:41:36

在 VMware Ubuntu 22.04 客户机上通过 Packer 和 Terraform 运行 cloud-init 两次

问题描述

1 个解决方案

解决方案1 0 已采纳 2022-06-27 07:41:36

解决方案1
0 已采纳 2022-06-27 07:41:36