Java MergeSort Binary files

Question

I have several sorted binary files which store information in some variable length format (meaning one of the segments contains the length of the variable length segment).

I need to merge them into one sorted file. I can do so with BufferedInputStream successfully. Nevertheless, it takes very long time on a mechanical disk. On a machine with SSD its much faster, as expected.

What bothers me is the fact that even on SSD, the CPU utilization is very low, and makes me suspect there's a way to improve the speed. I assume this happens because most of the time the CPU waits on the disk. I tried to increase the buffers to hundreds of MBs to no avail.

I have tried to use MemoryMapped buffer and file channel but it didn't improve the runtime.

Any ideas?

Edit: Using MemoryMappedByteBuffer failed because the merged file size is over 2 GB, which is the size limitation of MemoryMappedByteBuffer. But even before having merged the smaller files into GB files, I didn't notice an improvement in speed or CPU utilization.

Thanks

Answer 1

Perhaps you can compress the files better or is that not an option? If the bottleneck is I/O then reducing the amount is a good attack angle. http://www.oracle.com/technetwork/articles/java/compress-1565076.html