SSM send command to EC2 instance Failed

Question

I'm trying to use boto3 to run ssh commands on EC2 instances. I read this guide: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-remote-commands.html and I did everything of what they wrote there but I keep get error message:

>>>import boto3
>>> ec2 = boto3.client('ssm')
>>> a = ec2.send_command(InstanceIds=['i-0d5e16f6'], DocumentName='AWS-RunShellScript', Comment='abcdabcd', Parameters={"commands":["ifconfig"]})

output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 253, in _api_call
  return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 543, in _make_api_call
  raise error_class(parsed_response, operation_name)
  botocore.errorfactory.InvalidInstanceId: An error occurred (InvalidInstanceId) when calling the SendCommand operation: 

if I'm trying to send command with awscli I get the same problem:

aws ssm send-command --instance-ids "i-0d5e16f6" --document-name "AWS-RunShellScript" --comment "IP config" --parameters commands=ifconfig --output text

An error occurred (InvalidInstanceId) when calling the SendCommand operation:

some one know how to solve it?

Answer 1

This can happen when you don't have SSM agent installed on the instance you're trying to access. For a list of instances where you can run SSM commands, run:

aws ssm describe-instance-information --output text

From there, you can grab an instance ID and then run the send_command command with that instance.

Answer 2

As documented here in AWS' troubleshooting guide there are a range of possible causes for this error.

The accepted answer aws ssm describe-instance-information checks for instances which are both available, in a valid state and have the SSM agent installed, so that covers several of troubleshooting steps in one line (nice ;) ).

If you're using boto3 the same can be achieved with:

ssm.client.describe_instance_information()

I'm not certain whether it checks permissions but presume so. If your instance_id is missing from the list, you can ensure correct permissions by following the step by step here .

However, there is another cause (last but definitely not least as it's not obvious):

Freshly created instances take a little while to show up in the describe_instance_information list .

This is even after waiting for the instance to complete post-creation. So for example doing:

    # Key names are the same as the keyword arguments required by boto
    params = {
            'ImageId': image_id_to_use,
            'InstanceType': instance_type_to_launch,
            'MinCount': 1,
            'MaxCount': 1,
            'UserData': user_data_script,
            'SecurityGroups': ['your groups'],
            'KeyName': 'yourkeyname',
          }

    # Run the instance and wait for it to start
    reservation = ec2.client.run_instances(**params)
    instance = ec2.resource.Instance(reservation['Instances'][0]['InstanceId'])
    instance.wait_until_running()

    # Also wait status checks to complete
    waiter = ec2.client.get_waiter('instance_status_ok')
    waiter.wait(InstanceIds=[instance.id])

    # Apply the IAM roles required (this instance will need access to, e.g., S3)
    response = ec2.client.associate_iam_instance_profile(
        IamInstanceProfile={
            'Arn': 'your_arn',
            'Name': 'ApplicableRoleEGAdministratorAccess'
        },
        InstanceId=instance.id
    )

    print('Instance id just created:', instance.id)
    print('Instances in the SSM instances list right now:')
    print(ssm.client.describe_instance_information()['InstanceInformationList'])

Will highlight this problem (if present - it certainly was for me).

This may be due to the time taken to execute the UserData script (see this SO post for a possibly-related discussion on waiting for user data to complete ), but I can't tell (without more effort than I'm willing to take!) whether it's that, or just the time inherent in AWS updating its services database.

To solve this, I wrote a short waiter (with a timeout exception to handle other failure modes) that repeatedly called describe_instance_information() until the instance id showed up in the list.