尝试使用 boto3 为 EMR 获取实例类型和计数时出错

Question

Am trying to fetch EMR Core instance type, and Running instance counts. The following aws cli command works fine, and am trying to fetch it using boto3, using custom filter, but getting issue.

AWS CLI command:

aws emr describe-cluster --cluster-id 'j-xxxxxx' --profile=dev --query 'Cluster.InstanceGroups[?InstanceGroupType==`CORE`].[InstanceType, RunningInstanceCount]'

The response for above command:

[
    [
        "CORE", 
        "c4.8xlarge", 
        10
    ]
]

Am trying to use boto3 using a custom_filter, but getting errors with custom_filter.

cluster_id = 'j-xxxxx'
custom_filter = [{'Name':"ClusterId", 'Values': [cluster_id]},
                     {'Name': "InstanceGroupType", 'Values': 'CORE'}]
print(custom_filter)
response = emr_client.list_instance_groups(Filters = custom_filter)

The error message in EMR logs:
Traceback (most recent call last):
  File "/mnt/tmp/spark-ea1adb4d-ff6f-4bbd-8eae-39e89cf5ba47/boto3_emrec2_test.py", line 108, in <module>
    boto3_test()
  File "/mnt/tmp/spark-ea1adb4d-ff6f-4bbd-8eae-39e89cf5ba47/boto3_emrec2_test.py", line 66, in boto3_test
    response = emr_client.list_instance_groups(Filters = custom_filter)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 692, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 740, in _convert_to_request_dict
    api_params, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line 360, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Missing required parameter in input: "ClusterId"
Unknown parameter in input: "Filters", must be one of: ClusterId, Marker

Have also tried with tag:ClusterID, tag:Id, etc. but getting the same error. Appreciate any pointers.

Answer 1

I am able to get the instance_type and running instance count using the following code snippet in boto3:

emr_client = boto3.client("emr")
cluster_id = 'j-xxxxxx'
response = emr_client.list_instance_groups(ClusterId = cluster_id)
    core_info = list(filter(lambda m: 'CORE' in m["InstanceGroupType"], response["InstanceGroups"]))
    instance_type = core_info[0]['InstanceType']
    core_instance_count = core_info[0]['RunningInstanceCount']

And if you would need vCPUs for a given instance, the following will return the vCPUs:

instance_dtl = ec2_client.describe_instance_types(InstanceTypes=[instance_type])
current_vcpu_count = instance_dtl['InstanceTypes'][0]['VCpuInfo']['DefaultVCpus']

Hope it will be useful for someone.