How to create a stream.parallel ItemReader in spring-batch?

Question

How can I use the new java 8 streams.parallel api in spring-batch ItemReader ?

list.stream().parallel().forEach(file -> readLines(file));

Typically, an ItemReader reads all lines from a file up to a defined chunk size, then passes the chunked list to an itemwriter. And when the writer finished, the next chunk is read.

I want this process of reading+writing to be using the java8 stream.parallel api. Is that possible?

Answer 1

Within the reader itself, I'm not 100% sure that use case makes sense. The idea of the stream api is to stream from a source to a terminal operation. The contract for a Spring Batch ItemReader is to provide items one at a time which obviously isn't ideal. I think the closest you could get is to use the stream api to load the items into memory, but there is no way to stop it from reading between items or even chunks.

I've been looking at the idea of creating a Step implementation that uses a stream concept (either Java 8 streams or reactive streams via Reactor or RxJava) but that won't be available until the 3.1 release at the earliest (I'd be open to working it with you if you're interested). That would use a stream provider of some kind as the "reader", the intermediate functions serve the role "processor", and a terminal function serves as the "writer".