How to get the list of instances for AWS EMR?

Question

Why is the list for EC2 different from the EMR list?

EC2: https://aws.amazon.com/ec2/spot/pricing/

EMR: https://aws.amazon.com/emr/pricing/

Why are not all the types of instances from the EC2 available for EMR? How to get this special list?

Answer 1

In case your question is not about the amazon console
(then it would surely be closed as off-topic ):

As a programming solution, you are looking something like this: ( using python boto3 )

import boto3
client = boto3.client('emr')
for instance in client.list_instances():
  print("Instance[%s] %s"%(instance.id, instance.name))

Answer 2

This is what I use, although I'm not 100% sure it's accurate (because I couldn't find documentation to support some of my choices (-BoxUsage, etc.)).

It's worth looking through the responses from AWS in order to figure out what the different values are for different fields in the pricing client responses.

Use the following to get the list of responses:

   default_profile = boto3.session.Session(profile_name='default')
    # Only us-east-1 has the pricing API
    # - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/pricing.html
    pricing_client = default_profile.client('pricing', region_name='us-east-1')
    service_name = 'ElasticMapReduce'
    product_filters = [
        {'Type': 'TERM_MATCH', 'Field': 'location', 'Value': aws_region_name}
    ]
    response = pricing_client.get_products(
        ServiceCode=service_name,
        Filters=product_filters,
        MaxResults=100
    )
    response_list.append(response)

    num_prices = 100
    while 'NextToken' in response:
       # re-query to get next page

Once you've gotten the list of responses, you can then filter out the actual instance info:

  emr_prices = {}
  for response in response_list:
      for price_info_str in response['PriceList']:
          price_obj = json.loads(price_info_str)
          attributes = price_obj['product']['attributes']

          # Skip pricing info that doesn't specify a (EC2) instance type
          if 'instanceType' not in attributes:
              continue
          inst_type = attributes['instanceType']

          # AFAIK, Only usagetype attributes that contain the string '-BoxUsage' are the ones that contain the prices that we would use (empirical research)
          # Other examples of values are <REGION-CODE>-M3BoxUsage, <REGION-CODE>-M5BoxUsage, <REGION-CODE>-M7BoxUsage (no clue what that means.. )
          if '-BoxUsage' not in attributes['usagetype']:
              continue

          if 'OnDemand' not in price_obj['terms']:
              continue

          on_demand_info = price_obj['terms']['OnDemand']
          price_dim = list(list(on_demand_info.values())[0]['priceDimensions'].values())[0]
          emr_price = Decimal(price_dim['pricePerUnit']['USD'])

          emr_prices[inst_type] = emr_price

Realistically, it's straightforward enough to figure this out from the boto3 docs. In particular, the get_products documentation.