Spring Batch: Specifying chunk size for processing List of Lists

Question

I am working on List containing multiple Lists of records from database.

How should I specify chunk size for processing each sublist?

I do have chunk configuration in my Steps, which works on number of sublists, and not the contents of sublist.

My usecase is, each sublist can contain 5000-10000 records to be processed, and can have 4-5 such lists, do not want to process all this in one chunk.

Appreciate your help, Thanks in advance.

Answer 1

This is not possible. The chunk size (commit-interval) is the number of items in a chunk. If your item is a list (regardless of how many items in this list), the chunk size will be the number of lists to read/process/write in one transaction.

Answer 2

You can implement CompletionPolicy like

<batch:step id="step1"> 
<batch:tasklet> 
<batch:chunk reader="itemReader" processor="processor" 
writer="itemWriter" chunk-completion-policy="completionPolicy"> 
</batch:chunk> 
</batch:tasklet> 
</batch:step>

using CompletionPolicy you can control chunk size based on your list dynamically.

public class RandomChunkSizePolicy implements CompletionPolicy {
private int chunkSize;
private int totalProcessed;

public boolean isComplete(RepeatContext context) {          
return totalProcessed >= chunkSize;      
}

public boolean isComplete(RepeatContext context, RepeatStatus status) 
{          
    if (RepeatStatus.FINISHED == status) {              
        return true;          
    } else {              
        return isComplete(context);          
    }      
}

public RepeatContext start(RepeatContext context) {
          Random random = new Random();          
          chunkSize = random.nextInt(20);
          totalProcessed = 0;            
          System.out.println("The chunk size has been set to " + chunkSize);            
          return context;      
}

public void update(RepeatContext context) {          
    totalProcessed++;      
}