Loading files to RAM (static) for improving the read time

Question

I am writing a C program for a small application where I need to read huge data from a file into a buffer and I need to do some mathematical operations on this data in the buffer. Problem is that, lets assume I have a file of size around 7MB, the reading values from this buffer is taking much time thereby degrading the performance of my application. Is it possible to write my file data into the RAM to get better performance by reducing the read operation from the buffer?

Answer 1

Yes, pre-loading the file into memory can improve performance. Here's the code to load a file given a FILE *fpin

size_t size = _filelength( _fileno( fpin ) );
void *buffer = malloc( size );
if ( !buffer || fread( buffer, 1, size, fpin ) != size )
{
    fprintf( "file load failed" );
    exit( 1 );
}

Edit - loop optimization

Try replacing this

for(i = index; i < (index+window_size); i++) 
{ sum+= buffer[i]; }

with this

unsigned short *bufptr = &buffer[index];
unsigned short *endptr = &buffer[index+window_size];
for ( ; bufptr < endptr; bufptr++ ) 
    sum += *bufptr; 

Answer 2

Going a little different direction I'm going to suggest that you profile your application and see exactly where all the time is spent. With enough experience you can sometimes guess correctly but often the slow areas are non-intuitive. Guessing incorrectly means you spend time optimizing things that have little to no affect on the overall speed of your program.

My guess is that the loading time of your 7 MB file is not the primary issue here as on modern hardware file load times are much lower than you'd expect. For example, I quickly tested reading a 14 MB file and load times were around 6 ms. Assuming you are loading your file only once I wouldn't expect this to be an issue.

Depending on your version of VS2010 you can use the built-in profiling abilities. If not you can create your own simple profiling ability using the Windows QueryPerformanceCounter() function. Once you find what areas are the slowest then can you start to look at how to make them faster.