ReadFile FILE_FLAG_NO_BUFFERING how to read data between two sectors

Question

I´m using Windows ReadFile function to read a 4GB files sequentially, with FILE_FLAG_NO_BUFFERING. I´m using a buffer of 64K, and all works right, but the problem is that my data is cut between the end of the current buffer and the next read. For example, I have a sequence of 4 bytes float numbers, and when I arrive to the last float only three bytes are in the current buffer, and the next byte will come in the next read buffer. So how can I handle this? Should I track the number of bytes consumed to keep my last three bytes and then append the last one after read the next buffer? Or maybe copy the buffer to another and make the tracking of the floats there? But this doesn´t defeat the advantage of not reading with Windows cache? Thanks for any help.

Answer 1

I think that in the special case where you are on the boundary, you should copy the bits from the previous block and the bits from the next block side-by-side into a small scratch space, and read them from there in one piece. When you are not on the boundary, you need not use this scratch space.

Answer 2

One way or another, you have to do this accounting yourself, since unbuffered reads are required to be aligned.

But a better question is, why do you think you need to do this? Have you tried using std::ifstream to read your file? Modern processors and caches go a long way toward hiding (or, in effect, eliminating) the extra copy that you probably think buffered I/O performs.

Also, if you are reading sequentially, FILE_FLAG_NO_BUFFERING will inhibit the OS's read-ahead machinery. This will almost certainly cost you far more than you will gain from bypassing the OS buffers.

I suspect you will find that the simplest code will perform best for sequential reading of large files. That is pretty much the case modern systems are optimized for at every level...