使用AWS Cookbook InvalidInstanceIDNotFound的Chef EC2 EBS卷

Question

I'm using the Chef provided AWS cookbook (v2.7.2) to create and attach an EBS volume on an Ubuntu Server 14.04.02 LTS HVM ec2 instance during the initial bootstrapped chef-client run. I'm hosting my own chef server (12.1.0-1_amd64). Using knife-ec2, I create the aws instance and bootstrap the runlist via the following command:

knife ec2 server create --region us-east-1 --availability-zone us-east-1a --node-name my-server --run-list "role[my-role]" --image ami-d05e75b8 --flavor c3.large --ebs-size 16 --ebs-volume-type gp2 --subnet subnet-mySubnetId --associate-public-ip --server-connect-attribute public_ip_address --ssh-user ubuntu --ssh-key my-pem --identity-file /path/to/my-pem.pem -y

However, when the recipe executes that creates and attaches the volume, the volume successfully creates, but fails to attach to my instance. The following error keeps occuring:

Aws::EC2::Errors::InvalidInstanceIDNotFound: The instance ID 'i-36a93b84' does not exist

The following code is in my recipe (taken from the example on the cookbook page):

include_recipe 'aws'

aws_ebs_volume "backup_volume" do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  size 16
  device "/dev/sdi"
  availability_zone "us-east-1a"
  action [ :create, :attach ]
end

I verified 'i-36a93b84' is the id of the instance knife-ec2 creates and the chef-client is executing on, so it does exist. Also, I validated that I'm creating the volume in the same availability zone as the instance.

I've read some posts about AWS's Eventual Consistency , but have no idea how to handle this during a bootstrapped chef-client run, since the recipe runs immediately after the instance is created.

Is there a way to create an additional mount at the time knife-ec2 creates the aws server, or am I trying to attach a volume in an unorthodox manner?

Chef client output during the recipe:

Recipe: my-cookbook::backupvolume
      * aws_ebs_volume[backup_volume] action create[2015-07-17T11:57:05+00:00] INFO: Processing aws_ebs_volume[backup_volume] action create (my-cookbook::backupvolume line 49)
      [2015-07-17T11:57:05+00:00] DEBUG: Providers for generic aws_ebs_volume resource enabled on node include: [LWRP provider aws_ebs_volume from cookbook aws]
      [2015-07-17T11:57:05+00:00] DEBUG: Provider for action create on resource aws_ebs_volume[backup_volume] is LWRP provider aws_ebs_volume from cookbook aws
      [2015-07-17T11:57:05+00:00] DEBUG: Instance ID is i-36a93b84
      [2015-07-17T11:57:05+00:00] DEBUG: Instance's availability zone is us-east-1a
      [2015-07-17T11:57:05+00:00] DEBUG: Created new  volume vol-4842ecc4
      [2015-07-17T11:57:05+00:00] DEBUG: Volume is creating
      [2015-07-17T11:57:08+00:00] INFO: Volume vol-4842ecc4 is available
        - create a volume with id= size=16 availability_zone=us-east-1a and update the node data with created volume's id
        * aws_ebs_volume[backup_volume] action attach[2015-07-17T11:57:08+00:00] INFO: Processing aws_ebs_volume[backup_volume] action attach (my-cookbook::backupvolume line 49)
      [2015-07-17T11:57:08+00:00] DEBUG: Providers for generic aws_ebs_volume resource enabled on node include: [LWRP provider aws_ebs_volume from cookbook aws]
      [2015-07-17T11:57:08+00:00] DEBUG: Provider for action attach on resource aws_ebs_volume[backup_volume] is LWRP provider aws_ebs_volume from cookbook aws
      [2015-07-17T11:57:08+00:00] DEBUG: Attaching vol-4842ecc4 as /dev/sdi


        ================================================================================
        Error executing action `attach` on resource 'aws_ebs_volume[backup_volume]'
        ================================================================================

        Aws::EC2::Errors::InvalidInstanceIDNotFound
        -------------------------------------------
        The instance ID 'i-36a93b84' does not exist

        Cookbook Trace:
        ---------------
        /var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:212:in `attach_volume'
        /var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:70:in `block (2 levels) in class_from_file'
        /var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:68:in `block in class_from_file'

        Resource Declaration:
        ---------------------
        # In /var/chef/cache/cookbooks/my-cookbook/recipes/backupvolume.rb

         49:       aws_ebs_volume "backup_volume" do
         50:         aws_access_key aws['aws_access_key_id']
         51:         aws_secret_access_key aws['aws_secret_access_key']
         52:         size 16
         53:         device "/dev/sdi"
         54:         description "TEST BACKUP"
         55:         availability_zone "us-east-1a"
         56:         action [ :create, :attach ]
         57:       end
         58:

        Compiled Resource:
        ------------------
        # Declared in /var/chef/cache/cookbooks/my-cookbook/recipes/backupvolume.rb:49:in `from_file'

        aws_ebs_volume("backup_volume") do
          action [:create, :attach]
          updated true
          retries 0
          retry_delay 2
          default_guard_interpreter :default
          declared_type :aws_ebs_volume
          cookbook_name "my-cookbook"
          recipe_name "backupvolume"
          aws_access_key "********"
          aws_secret_access_key "********"
          size 16
          device "/dev/sdi"
          description "TEST BACKUP"
          availability_zone "us-east-1a"
          timeout 180
          volume_type "standard"
          piops 0
        end

Here is the stacktrace:

Aws::EC2::Errors::InvalidInstanceIDNotFound: aws_ebs_volume[backup_volume] (my-cookbook::backupvolume line 49) had an error: Aws::EC2::Errors::InvalidInstanceIDNotFound: The instance ID 'i-36a93b84' does not exist
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/plugins/param_conversion.rb:22:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/plugins/response_target.rb:18:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/request.rb:70:in `send_request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/base.rb:216:in `block (2 levels) in define_operation_methods'
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:212:in `attach_volume'
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:70:in `block (2 levels) in class_from_file'
/opt/chef/embedded/apps/chef/lib/chef/mixin/why_run.rb:52:in `call'
/opt/chef/embedded/apps/chef/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/chef/embedded/apps/chef/lib/chef/provider.rb:175:in `converge_by'
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:68:in `block in class_from_file'

Answer 1

The problem was due to my own foolishnes.

The AWS credentials in the databag were for an old decomissioned AWS account but the AWS credentials used to create the instance w/ knife were for the current AWS account.

Once they correct keys were put in the databag, the volume successfully created and attached!

Hope this helps someone in the future...