Fastest way to split Amazon DynamoDB ItemCollection into small pieces

Question

I have a need to build a lambda function that pulls back several thousand items from DynamoDB and processes them before Api Gateway times out (29 seconds). I thought the best way to do this would be to break the collection into smaller chunks and spread them across multiple threads to process in parallel.

The problem is, doing it the way that I am currently attempting is taking ~25 seconds just to split the collection. Is there a better way to go about this in order to process much more quickly?

Code:

public static List<List<Item>> partitionList(ItemCollection<QueryOutcome> items) {
    final int partitionSize = 20;
    List<List<Item>> partitioned = new LinkedList<List<Item>>();
    List<Item> itemList = new ArrayList<Item>();
    for(Item item : items) {
        itemList.add(item);
    }

    for (int i = 0; i < itemList.size(); i += partitionSize) {
        partitioned.add(itemList.subList(i, Math.min(i + partitionSize, itemList.size())));
    }
    return partitioned;
}

Answer 1

My Java is a little rusty, so bear with me on the details, but it seems like you are doing a fair amount of extra work here.

public static List<List<Item>> partitionList(ItemCollection<QueryOutcome> items) {
    final int partitionSize = 20;
    List<List<Item>> partitioned = new LinkedList<List<Item>>();
    List<Item> itemList = new ArrayList<Item>();
    for(Item item : items) {
        itemList.add(item);
        if (itemList.size() == partitionSize) {
            partitioned.add(itemList);
            itemList = new ArrayList<Item>();
        }
    }

    if (itemList.size() != 0) {
        partitioned.add(itemList);
    }

    return partitioned;
}

I'd be curious about how you are populating the initial list here, as that may be the real place to do this work.

Also, you probably should consider something that is more asynchronous for this type of work.