扫描 10GB 的大型 Amazon DynamoDB 数据

Question

The following code works for me but it takes 19 minutes for 1 API request to return a result. An optimized result would be appreciated. I would not like to go for segments because then I will have to do thread management.

 dynamodb = boto3.resource('dynamodb', region_name='us-west-2', endpoint_url="http://localhost:8000") table = dynamodb.Table('Movies') fe = Key('year').between(1950, 1959) pe = "#yr, title, info.rating" # Expression Attribute Names for Projection Expression only. ean = { "#yr": "year", } esk = None response = table.scan( FilterExpression=fe, ProjectionExpression=pe, ExpressionAttributeNames=ean ) for i in response['Items']: print(json.dumps(i, cls=DecimalEncoder)) // As long as LastEvaluatedKey is in response it means there are still items from the query related to the data while 'LastEvaluatedKey' in response: response = table.scan( ProjectionExpression=pe, FilterExpression=fe, ExpressionAttributeNames= ean, ExclusiveStartKey=response['LastEvaluatedKey'] ) for i in response['Items']: print(json.dumps(i, cls=DecimalEncoder))

Answer 1

Because it is searching across all partitions, the scan operation can be very slow. You wont be able to "tune" this query like you might if you were working with a relational database.

In order to best help you, I will need to know more about your access pattern (get movies by year?) and what your table currently looks like (what are your partition keys/sort keys, other attributes, etc).

Answer 2

Unfortunately, scan is slow by nature. There is no way to optimize at the code level except for redesigning the table to optimize for this access pattern.