Spring批处理多线程ItemWriter

Question

I am quite new to Spring batch framework.

I am currently writing a batch with a reader and a writer.

Reader reads from Db and writer writes to a flat file. The number of records are 1 million. Writing to file takes a lot of time and I want to improve on that.

What is the best way I can achieve multithreading in writer so that write() method runs in parallel?

Note: In @BeforeStep and @AfterStep callbacks, I am writing header and footer of the file. write() method writes the records to file.

EDIT:

I have found out that, writing to file isn't taking much time but one of our internal method which does some sort of decryption takes about 500ms for 1 record. And we have 1 million such records.

Can I improve the performance by doing decryption in multiple threads? I am not able to understand how to improve from here on.

Answer 1

This is not really a Spring specific question. Typically what people do is they implement some kind of streaming. As in you don't read the entire query then write everything, but instead read little bits one after another, and then pass each bit onto the writer so it can already begin writing before you have finished reading. This is quicker and enables you also to not use as much memory. For instance if you had 10GB of data to read and write, you could split it up into 10MB queries, instead of reading the whole 10GB. You should read up on streams. Parallel writes to the same file however will result in no benefit, or reduced performance. You should also be careful not to start too many threads, again this will reduce performance, and unless your threads are really cheap I don't recommend really making more than 2, as some have already mentioned that the performance of most apps are I/O bound, and there is no getting around that, only mitigating effects by buffering/streaming/caching not blocking threads, or anything else you can do in your app.