从Spring Batch分区步骤的Writer启动Runnable

Question

I have a Spring batch job consisting of a partitioned step and partitioned step is doing processing in chunks.

Can I further launch new threads ( implementing Runnable ) from method, public void write(List<? extends VO> itemsToWrite) ?

Basically, writer here writes indices using Lucene and since writer has a List of chunk-size items, I thought to divide that List into segments and pass each segment to a new Runnable .

Is that a good approach?

I coded a sample and it works most of the times but gets stuck few times.

Are there any thing that I need to worry about? OR is there something inbuilt in spring batch to achieve this?

I don't want write to happen by a single thread for whole chunk. I wish to further divide up chunk.

Lucene IndexWriter is thread safe and a approach is listed here

Sample Code - Writer gets a List of items for which I open threads from thread pool? Will there be any concern even if I wait for pool to terminate for a chunk,

@Override
    public void write(List<? extends IndexerInputVO> inputItems) throws Exception {


        int docsPerThread = Constants.NUMBER_OF_DOCS_PER_INDEX_WRITER_THREADS;
        int docSize = inputItems.size();
        int remainder = docSize%docsPerThread;
        int poolSize = docSize/docsPerThread;

        ExecutorService executor = Executors.newFixedThreadPool(poolSize+1);


        int fromIndex=0;
        int toIndex = docsPerThread;

        if(docSize < docsPerThread){
            executor.submit(new IndexWriterRunnable(this.luceneObjects,service,inputItems));
        }else{
            for(int i=1;i<=poolSize;i++){
                executor.submit(new IndexWriterRunnable(this.luceneObjects,service,inputItems.subList(fromIndex, toIndex)));
                fromIndex+=docsPerThread;
                toIndex+=docsPerThread;
            }

            if(remainder != 0){
                toIndex=docSize;
                executor.submit(new IndexWriterRunnable(this.luceneObjects,service,inputItems.subList(fromIndex, toIndex)));
            }
        }

        executor.shutdown();

        while(executor.isTerminated()){
            ;
        }

Answer 1

I'm not sure that launching new Threads in writer it's the good idea. These threads are out of scope of spring batch framework, so you will need to implement shutdown and cancellation policy for above. If processing of one segments will fail it can lead to fail entire queue.

As alternate approach I can suggest to promote your custom segments of list from writer to next step as described in official docs passingDataToFutureSteps