Calculate sha-1 or CRC32 checksum for big files > 400mb

Question

I have a newbie question, I've never read files bigger than 4Mb with function like fopen, ReadFile(WINAPI). My question is What is the best way to read a big file like 400Mb load blocks of 512 bits into memory?

Thanks

Answer 1

First of all - at first instance - you may not want to read that small - a few kBytes is better when it comes from disk. In your case - a simple implementation could read 16k - and then cycle through those 32 times for your 512 bytes operations.

In reality that is not so important -as the OS is clever and generally will second guess you - and read more anyway. So a naive implementation where you simply do a

do 
{ 
  bResult = ReadFile(hFile, &inBuffer, nBytesToRead, &nBytesRead, NULL); 
  do-something-with-nBytesRead; 
} while(!bResult &&  nBytesRead != 0); 
if (nBytesRead) error..; 

sort of thing is fine.

If you know you are always going to have such (rather large) files, performance is important and there is other use of those files too - then consider 'memory mapping' the file - ie open it in such a way that the entire file appears in virtual memory. Have a look at http://msdn.microsoft.com/en-us/library/windows/desktop/aa366556(v=vs.85).aspx for that.

Answer 2

If you read large files, you often want to turn off file buffering. Because if you don't, you fill the file cache of your system with these large files even though you just read or write them once and don't use them any longer. As a consequence, files that are more frequently accessed are no longer cache and your system will be slow for a certain time.

The article File Buffering describes how to access files without buffering. Unfortunately, direct file access isn't as easy as it could be in Windows.

Having said that, 400 MByte isn't usually a problem. I wouldn't use direct file access unless the files are considerably larger than 1 GByte.