boto3 dynamodb batch_get_item in index not working

Question

I have a working boto3 dynamodb query below:

from boto3.dynamodb.conditions import Key
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('gpc')
    video_id = 25
    response = table.query(
        IndexName='skuindex',
        KeyConditionExpression=Key('sku').eq('98123')
    )
    print(response)

Which will look inside my table 'gpc', inside the index 'skuindex' for rows where sku=98123. It returns one result correctly.

I am trying to replicate this query in a batch_get_item command, but am unable to query for the sku field, and can't figure out how to specify the index.

The following batch_get_item code works:

boto3.resource('dynamodb')
dynamo = boto3.client("dynamodb", region_name="us-east-1")
serializer = boto3.dynamodb.types.TypeSerializer().serialize

test_keys = [] 
test_keys.append({
    "upc": serializer("8122222"), 
    #"sku": serializer("98123"), 
    "source_store": serializer("itemname_itemname")
}) 
test_response = dynamo.batch_get_item(
    RequestItems={
        "gpc": {'Keys': test_keys}
    }
)
print("len(test_response['Responses']['gpc']) = ", len(test_response['Responses']['gpc']))

Is there some way I can specify inside this above to use the 'skuindex' index? Is that even possible with the batch_get_item command? If i uncomment the sku line, it leads to an error:

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the BatchGetItem operation: The provided key element does not match the schema

Because the sku field is only accessible in the 'skuindex' index apparently. I've tried adding index specific code to the batch_get_item query with no success.

How can I tell my batch_get_item to look inside the 'skuindex' index of my 'gpc' table?

Answer 1

Rafael Almeida's answer, saying that BatchGetItem (and also GetItem ) cannot read using an index is correct. He said it is a DynamoDB limitation , but I think it's interesting to understand why using an index simply doesn't fit the GetItem or BatchGetItem request API, but does fit Query :

One important difference between the Query request and GetItem / BatchGetItem requests is that Query API is designed to be able to return an unbounded number of responses. Query is therefore designed around the notion of paging and its ability to get back any number of items. In contrast, GetItem will always return one item (or none), while BatchGetItem can return more - but they will all be stuck in a single response.

Now, the thing is that the nature of an index is that you can never know how many items will actually match your lookup sku=98123 . Your application assumes it will be just one, but nothing actually prevents it from storing a million items, all having the same sku value. An item's key needs to be unique, but nothing guarantees that a non-key attribute is unique.

So, if DynamoDB were to allow a GetItem with an index lookup sku=98123 , it would need to somehow be able to return - potentially - either one or a million different results. Basically, Amazon would have needed to add support for Limit and paging to GetItem - but there is already a request that had exactly such support built-in, and that was Query . This is why you need to do index queries through Query .

You are right - it means you can do multiple such reads in one request. There is no BatchQuery request. But to be honest, a BatchGetItem didn't need to exist either - users should have been able to send a stream of separate GetItem or Query requests on the same connection. This could have just as efficient as today's BatchGetItem if DynamoDB used HTTP/2 - which finally added proper support for request pipelining (and out-of-order responses). Maybe one day DynamoDB will support HTTP/2 (as far as I know, it still doesn't).

Answer 2

Is there some way I can specify inside this above to use the 'skuindex' index?

Unfortunately, you can't, and this is not a boto3 limitation but a DynamoDB limitation .