Retrieving Data from DynamoDB using AWSSDK.net (core version 3.3.5 and AWSSDK.DynamoDBv2 version 3.3.1)

Question

I am trying to retrieve the data from Dynamo DB based on a search criteria using Scan request. I am following the steps mentioned in http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelDotNetScanning.html " page. My DynamoDB table contains more than 1 million records. I know that by using the do-while loop and ExclusiveStartKey we can fetch the records from dynamo DB. but in my case I can not wait till search process is complete as this will hang my angularJS UI. instead I want to progressively load the data with out waiting for the search process to complete. How we can do that.?

sample request:

var lastEvaluatedKey = new Dictionary<string, AttributeValue>(); ;
    AmazonDynamoDBClient amazonDynamoDbClient= new AmazonDynamoDBClient()
        var filterExpression = "#aws_s3_bucket = :v_aws_s3_bucket and contains(#aws_s3_key,:v_aws_s3_key)";
        var projectExpression = "#aws_s3_key,filename,#region,aws_s3_bucket,#projecttype,folder,#siteid,locationname,createdon,modifiedon";



        do
        {   
        var request = new ScanRequest
            {
                TableName = "Test1",
                ExclusiveStartKey=lastEvaluatedKey,

                FilterExpression = filterExpression,
                ExpressionAttributeNames = new Dictionary<string, string>
                            {
                              { "#region", "region" },
                              { "#siteid", "siteid" },
                              { "#projecttype", "projecttype" },
                              { "#aws_s3_key", "aws_s3_key" },
                              { "#aws_s3_bucket", "aws_s3_bucket" }                       
                            },
                ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
                            {":v_aws_s3_bucket", new AttributeValue { S =  "sampleBucket"}},
                            {":v_aws_s3_key", new AttributeValue { S =  "92226"}}
                            },

                ConsistentRead = true,
                ProjectionExpression = projectExpression
            };

        response = amazonDynamoDbClient.Scan(request); 
    lastEvaluatedKey = response.LastEvaluatedKey;}while(lastEvaluatedKey!=null && lastEvaluatedKey.count()!=0)

I tried executing the above request with out using do-while loop and saved the ExclusiveStartKey for the next request it throws error the "The provided starting key is invalid: One or more parameter values were invalid: Null attribute value types must have the value of true".

any help on this issue will be helpful...

Answer 1

The error you're getting appears to be because you're setting ExclusiveStartKey on the request without setting any values for its parameters. Notice how you aren't updating request.ExclusiveStartKey after you get your response. Obviously, if you don't do that, the scan won't know where to pick up again when you hit your limit. See below.

AmazonDynamoDBClient amazonDynamoDbClient= new AmazonDynamoDBClient()
var filterExpression = "#aws_s3_bucket = :v_aws_s3_bucket and contains(#aws_s3_key,:v_aws_s3_key)";
var projectExpression = "#aws_s3_key,filename,#region,aws_s3_bucket,#projecttype,folder,#siteid,locationname,createdon,modifiedon";
ScanRequest request = new ScanRequest
        {
            TableName = "Test1",
            FilterExpression = filterExpression,
            ExpressionAttributeNames = new Dictionary<string, string>
                        {
                          { "#region", "region" },
                          { "#siteid", "siteid" },
                          { "#projecttype", "projecttype" },
                          { "#aws_s3_key", "aws_s3_key" },
                          { "#aws_s3_bucket", "aws_s3_bucket" }                       
                        },
            ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
                        {":v_aws_s3_bucket", new AttributeValue { S =  "sampleBucket"}},
                        {":v_aws_s3_key", new AttributeValue { S =  "92226"}}
                        },

            ConsistentRead = true,
            ProjectionExpression = projectExpression
        };

    do
    {   
        response = amazonDynamoDbClient.Scan(request); 
        request.ExclusiveStartKey = response.LastEvaluatedKey;
    } while (response.lastEvaluatedKey.Count != 0);