Concurrent processing of multiple files in java

Question

My requirement is like this. There is a central repository where user will place multiple xml files, our directory watcher service(simple java class) will pick up the xml files and process it.

In this scenario how can we go for concurrent processing? Should we go for one-by-one xml processing or concurrent processing?

Normally in live projects which approach should we take?

Answer 1

In this scenario how can we go for concurrent processing?

The directory watcher service could pick new files and place them in a queue where multiple threads pick up files and action them as needed.

Should we go for one by one xml processing or concurrent processing?

This depends on how quickly you want the processing done and other factors eg Will the processing be separate throughout or will the processing need access to other already currently files, Will you at some point need to wait for other resources?

Normally in live projects which approach should we take?

Concurrent processing might be faster but may require a little more memory

Answer 2

In this scenario by far and away your biggest cost will be IO, unless you're doing something with the data that is equally expensive (say talking to another service)

You may find that you spawn many threads and most of them dont do anything as you're processing the loaded (in memory) files quicker than you're actually loading them.

I'd suggest you see if you actually need to do this first, if you'll actually gain anything by implementing a concurrent approach. You may find that you dont gain anything meaningful in terms of throughput, but you have gained extra code complexity.