使用PagingAndSortingRepository进行Spring批处理步骤分区

Question

I am trying to configure Spring Batch Steps for partitioning. The nice sample found here shows a partition about "id range", but I don't know where to start for a "data page" range.

In my sequential step, I have :

• reader : a RepositoryItemReader using a PagingAndSortingRepository
• processor : a data converter
• writer : a RepositoryItemWriter using a CrudRepository
• chunck : 5
• listener : a StepListener

return stepBuilderFactory.get("stepApplicationForm") .<OldApplicationForm, NewApplicationForm>chunk(5) .reader(reader).processor(processor).writer(writer) .listener(listener).build();

As I have understood, for partitionning, I have to create a partitioner, then I have a "parent" step that tells to use the partitioner with the child step, then the "child" step with a reader aware of the "pagination" parameters.

For the TaskExecutor, I think that the ThreadPoolTaskExecutor will fit.

What is the good way to implement/configure a paritioning based on data "pages" ? And what are the threading caveeats I should check ?

Thanks :)

Answer 1

Each partition has its own item reader and item writer instances. Your partition implementation will find min max values of a data load. Using your own logic you can create min and max values in the execution context. While querying the data base you can make use of these to handle specific slice of the data so that no concurrency issues takes place.

@Bean
public Step myMasterStep() {
    return  stepBuilderFactory.get("myMasterStep")
            .partitioner("mySlaveWorker", myPartitioner())
            .partitionHandler(myPartitionHandler()).build();
}


@Bean
    public Step mySlaveWorker() {
        return stepBuilderFactory
                .get("mySlaveWorker")
                .<OldApplicationForm, NewApplicationForm> chunk(5)
                .faultTolerant()
                .listener(MyStepListener())
                .skip(DataAccessException.class)
                .skip(FatalStepExecutionException.class)
                .skip(Exception.class)
                .skipLimit(75)
                .noRollback(DataAccessException.class)
                .noRollback(FatalStepExecutionException.class)
                .noRollback(Exception.class)
                .reader(myDataItemReader())
                .writer(myDataItemWriter()).build();
    }

@Bean
@StepScope
public MyDataItemReader myDataItemReader(
        @Value("#{stepExecutionContext[minId]}") Long minId,
        @Value("#{stepExecutionContext[maxId]}") Long maxId) {
    MyDataItemReader myDataItemReader = new MyDataItemReader();
    myDataItemReader.setPageSize(100);
    myDataItemReader.setMinId(minId);
    myDataItemReader.setMaxId(maxId);
    return myDataItemReader;
}

@Bean
@StepScope
public MyDataItemWriter myDataItemWriter() {
    return new MyDataItemWriter();
}

@Bean
@StepScope
public MyPartitioner myPartitioner() {
    MyPartitioner myPartitioner = new MyPartitioner();
    myPartitioner.setDataSource(dataSource);
return myPartitioner;
}

public class MyStepListener implements SkipListener<OldApplicationForm, NewApplicationForm> {

private static final Logger LOGGER = LoggerFactory.getLogger(MyStepListener.class);


public void onSkipInProcess(OldApplicationForm item, Throwable t) {
    LOGGER.error("onSkipInProcess" + t.getMessage());
}

public void onSkipInRead(Throwable t) {
    LOGGER.error("onSkipInRead " + t.getMessage());
}


public void onSkipInWrite(NewApplicationForm item, Throwable t) {
    //logs
    LOGGER.error("In MyStepListener --> onSkipInProcess" + t.getMessage());
}

}