How to organize a Bento - Veewee - Vagrant project

Question

I'm trying to use Bento, Veewee and Vagrant to automate the process of provisioning VMs. This approach has a lot of promise for simplifying development, dev-testing and QA processes.

(For those who haven't encountered these -- Vagrant makes it easy to quickly set up / tear down VMs in Virtual Box, VMWare, EC2 etc.; Veewee simplifies the process of building base boxes for Veewee; and Bento simplifies the process of defining base boxes and then automating the operation of Veewee).

The problem is, the interrelationship of Bento, Veewee and Vagrant is confusing. Many definition files, and some operations, are duplicated across layers and it isn't clear which layer should get the updates that're specific to my project. The Ruby environment also complicates this quite a lot -- seems like Bento commands all need to be run from the Bento directory, so boxes & other files end up in illogical places.

Can anyone suggest a file layout & workflow for using this combination of tools?

Answer 1

I'm responding to my own question to capture what I discovered.

---

There's a way to make Bento, Veewee, Vagrant and Ruby work together in a semi-sensible way, but it's complicated. It's better to use the latest version of Bento that replaces Veewee with the Packer tool.

Packer does more or less what Veewee did, but made some great improvements including:

1. Each Packer box definition is in a single JSON file that refers to a flat list of script files -- no longer three layers of files with symlinks; easier to understand and better for source code control.

2. The Packer binaries are OS-native executables (no longer in Ruby). You deploy them by putting them somewhere on your path (like /usr/local/bin on Linux). Packer avoids the Ruby craziness.

---

Here is a simple workflow:

1. Install VirtualBox. Install Bento. Bento will also install Vagrant as a Ruby gem; but we won't use that instance of Vagrant. Install Vagrant separately using its native installer for your operating system.

2. Choose a name for your project to use in dir & file names. (For this example, I'm using "CUSTOM"). Create an empty sub-directory within the bento / packer / scripts directory. I did mkdir CUSTOM .

3. Review the .JSON files in .../ bento / packer. You can alter one in-place, or it's probably wiser to copy one & alter that. I did cp centos-6.4-x86_64.json centos-6.4-x86_64-CUSTOM.json .

4. Edit the "builders" section of that file to change the the VM's disk size, RAM size, ssh port etc.

5. Edit the "provisioners" section, "output" line to provide a meaningful name for the box file to be created. (For this example, I'm using "BOXFILE"). Note that you can embed {{timestamp}} to add a Unix timestamp to the file name -- helps to make the file name unique.

6. Edit the "provisioners" section of that file to refer to any scripts (in order) you want to run in the Packer build sequence. This would typically be a combination of the scripts already provided by Packer, plus whatever other scripts you build for your custom purposes.

The approach I used looked like this:

"provisioners": [
{
  "execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
  "scripts": [
    "scripts/common/sshd.sh",
    "scripts/common/vagrant.sh",
    "scripts/common/vmtools.sh",
    "scripts/CUSTOM/reso_1024x768.sh",
    "scripts/CUSTOM/prompt_cwd.sh",
    "scripts/CUSTOM/standard_dirs.sh",
    "scripts/CUSTOM/standard_utils.sh",
    "scripts/CUSTOM/python27.sh",
    "scripts/CUSTOM/supervisord.sh",
    "scripts/CUSTOM/ruby.sh",
    "scripts/centos/cleanup.sh",
    "scripts/common/minimize.sh"
  ],
  "type": "shell"
}

Note my custom scripts all appear before the cleanup and minimize steps -- this is important.

1. From the bento / packer directory, run packer build -only=virtualbox centos-6.4-x86_64-CUSTOM.json . If disk images (ISOs) for VirtualBox guest add-ons and base OS haven't been downloaded before, they will be now; these files are big so this step can take a while.

2. Packer uses VirtualBox to create a temporary VM, and runs all your specified scripts on it. When this process is done, a copy of the VM is captured in the bento / builds / visualbox directory using the BOXFILE name you specified above, and Packer discards the VirtualBox VM.

3. Register the box you just built with Vagrant. Using the separately installed version of Vagrant (eg you may have to specify /usr/local/bin/vagrant or whatever's appropriate for your OS), do vagrant box add CUSTOM path/to/BOXFILE . This creates a new sub-directory in ~/ .vagrant.d / boxes / CUSTOM / VirtualBox with the files Vagrant needs to create VMs from this box definition.

4. Create a Vagrant project directory and cd into it. I did mkdir ~/test; cd ~/test mkdir ~/test; cd ~/test .

5. Initialize a Vagrant project using this box: vagrant init CUSTOM

6. Start the VM: vagrant up

7. Connect to the VM: vagrant ssh

At this point, you're using Vagrant normally and Bento / Packer are no longer involved -- their combined job is just to create the basebox, and once that's been registered with Vagrant, Vagrant stands alone.

---

There are lots of other interesting things to look at re: Vagrant, like the special vagrant-aws plugin, and how to use it with Chef, Puppet or Ansible ... but I will stop here since my original question is answered.

Answer 2

It sounds like you're wanting to apply Bento, Vagrant, and/or Veewee to provision & perform QA testing on boxes. Thanks to the folks & community behind Chef , a tool already exists to do exactly what you are trying to do!

It's called test-kitchen . You can use it with the prebuilt Chef Bento Boxes which OpsCode/Chef maintains and builds with Packer. No need to build Vagrant boxes or VM images from scratch with VeeWee or Packer if you just want to get to Provisioning / Testing.

Test Kitchen is a test harness tool to execute provisioning code (in Chef , Ansible , Puppet , SaltStack , or Bash / Powershell via bootstrap.sh or bootstrap.ps1 ) on one or more platforms in isolation. It uses a driver plugin architecture which lets you plug it into various cloud providers and virtualization technologies such as Amazon EC2 , Blue Box , CloudStack , Digital Ocean , Rackspace , OpenStack , Vagrant , Docker , LXC containers , VMWare VSphere , Google Compute Engine , Azure , and many more. There are also "busser" testing plugins which allow pluggable support for multiple testing frameworks including Bats , shUnit2 , RSpec , Serverspec , Shpec , Tox , Nose2 , MiniTest , Cucumber , AnsibleSpec , with others being created.

For Chef, Puppet, and Ansible workflows, cookbook dependency resolver tools such as Berkshelf , Librarian-Chef , Librarian-Puppet , Ansible Galaxy , Librarian-Ansible are supported or you can simply have a cookbooks/ directory and Test Kitchen will know what to do. If you are already using Chef community cookbooks, Test Kitchen integration tests are already included in many such as the MySQL, nginx, Chef Server, and runit cookbooks.

It's pretty flexible and for your use case, you can probably make it do what you want by choosing the right set of gem plugins. Because it came out of the Chef community, you'll probably have the best experience using it with Chef cookbooks, Vagrant driver, and either ServerSpec or Bats testing plugins. However, it is gaining traction as a CI & QA testing tool on it's own with the many community plugins that are available.

If you're interested in seeing what's out there, just search RubyGems for kitchen- or busser- for more options.

If you're new to Ruby, Chef, or all of this in general... an easy way to get started is to install the ChefDK which includes test-kitchen , knife , Berkshelf and the basic / Chef.io-supported testing tools.

Once you have that, and if you're feeling adventurous or want to experiment with alternative pluggable provisioners, drivers, and busser plugins, you can install any of these into the ChefDK Ruby environment (example for Mac OS X / Unix) by using these commands:

$(/opt/chefdk/bin/chef shell-init $(basename $SHELL))
sudo /opt/chefdk/embedded/bin/gem install $kitchen_plugin_gem_name_here

If you want to play around with a demo repository that I've created for testing Ansible playbooks / roles, check out Ansible TDD