How to read/write at Maximum Speed from Hard Disk.Multi Threaded program I coded cannot go above 15 mb/ sec

Question

I have a 5 gb 256 Files in csv which I need to read at optimum speed and then write back data in Binary form .

I made following arrangments to achieve it :-

For each file, there is one corresponding thread. Am using C function fscanf,fwrite. But in Resource Monitor,it shows not more then 12 MB/ Sec of Hard Disk and 100 % Acitve Highest Time. Google says HardDisk can read/write till 100 MB/Sec. Machine Configuration is :- Intel i7 Core 3.4. Has 8 Cores.

Please give me your prespective. My aim to complete this process within 1 Min .

** Using One Thread it took me 12 Mins**

Answer 1

If all the files reside on the same disk, using multiple threads is likely to be counter-productive. If you read from many files in parallel, the HDD heads will keep moving back and forth between different areas of the disk, drastically reducing throughput.

I would measure how long it takes a built-in OS utility to read the files (on Unix, something like dd or cat into /dev/null ) and then use that as a baseline, bearing in mind that you also need to write stuff back . Writing can be costly both in terms of throughput and seek times.

I would then come up with a single-threaded implementation that reads and writes data in large chunks, and see whether I can get it to perform similarly the OS tools.

PS If you have 5GB of data and your HDD's top raw throughput is 100MB, and you also need to write the converted data back onto the same disk, you goal of 1 minute is not realistic.