无法将所有数据写入dynamodb表？

Question

I am trying to insert csv file data into dynamodb but I am able to write only 1537 records into my dynamodb table .

Coding

 try {

        S3EventNotificationRecord record = s3event.getRecords().get(0);
        String srcBucket = record.getS3().getBucket().getName();
        String srcKey = record.getS3().getObject().getKey().replace('+', ' ');  
        srcKey = URLDecoder.decode(srcKey, "UTF-8");
        AmazonS3 s3Client = new AmazonS3Client();
        S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
        statusReport.setFileSize(s3Object.getObjectMetadata().getContentLength());


        BufferedReader br = new BufferedReader(new InputStreamReader(s3Object.getObjectContent())); 
        CSVReader reader = new CSVReader(br);

        AmazonDynamoDB dynamoDBClient = new AmazonDynamoDBClient();

        dynamoDBClient.setRegion(AWS_REGION);
        DynamoDB dynamoDB = new DynamoDB(dynamoDBClient);
        TableWriteItems energyDataTableWriteItems = new TableWriteItems(DYNAMO_TABLE_NAME);
        List<Item> itemList = new ArrayList<Item>();
        String[] nextLine;


        while ((nextLine = reader.readNext()) != null) {
            Item newItem = helper.parseIt(nextLine);
            itemList.add(newItem);
        }
for (List<Item> partition : Lists.partition(itemList, 25)) {
            energyDataTableWriteItems.withItemsToPut(partition);
            BatchWriteItemOutcome outcome = dynamoDB.batchWriteItem(energyDataTableWriteItems);

 do {
        Map<String, List<WriteRequest>> unprocessedItems = outcome.getUnprocessedItems();

        if (outcome.getUnprocessedItems().size() > 0) {
                    logger.log("Retrieving the unprocessed " + String.valueOf(outcome.getUnprocessedItems().size())
                            + " items.");
                    outcome = dynamoDB.batchWriteItemUnprocessed(unprocessedItems);
                }

        } while (outcome.getUnprocessedItems().size() > 0);*/
}
 logger.log("Load finish in " + String.valueOf(System.currentTimeMillis() - startTime) + "ms");

        reader.close();
        br.close();
        s3Object.close();

        statusReport.setStatus(true);
    } catch (Exception ex) {
        logger.log(ex.getMessage());
    }

    statusReport.setExecutiongTime(System.currentTimeMillis() - startTime);
    return statusReport;
}

Can I know the reason why it is behaving like this. I have tried with more than 5 tables but I am getting the same result.

Answer 1

You're probably running into a throughput limitation:

From the docs :

The BatchWriteItem operation puts or deletes multiple items in one or more tables. A single call to BatchWriteItem can write up to 16 MB of data, which can comprise as many as 25 put or delete requests. Individual items to be written can be as large as 400 KB.

Also:

If DynamoDB returns any unprocessed items, you should retry the batch operation on those items.

While you are partitioning your writes into 25-item chunks, that's not the only limitation. If you look at the JavaDoc for BatchWriteItemOutcome you'll see the function getUnprocessedItems() .

Lastly, you don't show your catch block, but can we assume that it actually does something?