如何确定将大量文件复制到外部共享文件夹的并行Java线程的理想数量？

Question

What should be the ideal number of parallel java threads for copying a large set of files from a quad core linux box to an external shared folder? I can see that with a single thread it is taking a hell lot of time to move the files one by one. Multiple threads is improving the copy performance, but I don't know what should be the exact number of threads.

I am using Java executor service to create the thread pool.

Answer 1

Just try different variants and measure performance. There are also such bottlenecks as HDD and network speed, so I guess there is no definite answer.

Answer 2

Absolutely you should work this out by testing. As Dmitry points out, there are many factors involved. In fact, CPU is almost certainly not the bottleneck with an IO- and network-based operation. You'll probably find things level off and start to get worse before you have too many threads, but if you want to minimise testing, use a graph of a few results to interpolate a good guess for the ideal value.

To get a better understanding of what's taking all the time, use metrics tools to measure utilisation of your resources - in this case that should include network interface traffic and disk queue length.