Does reading via O_DIRECT flush dirty pages first?

Question

Assume I have a simple program that does the following:

fdWrite = open("file", O_WRONLY);
fdRead = open("file", O_RDONLY | O_DIRECT);
writeBuffer = <some data>;
write(fdWrite, writeBuffer);
readBuffer = read(fdRead, sizeof(writeBuffer));

Will I be guaranteed that readBuffer == writeBuffer ? (Given no other fd to this file is currently open, obviously)

Some simple testing on Linux seems to suggest that yes, the dirty pages from the write call will be flushed to disk before reading via O_DIRECT , but I can't seem to find any mention of this scenario anywhere. For all I know, it could be a complete coincidence that it worked, and I've no clue what would happen on other POSIX-like platforms. I'd like to get some "hard evidence" on this at least.

Why are you doing this?

It's in the context of an application distributing large files that get cached. Once a new part of the file is received, I'd like to verify the new part. I see two advantages using O_DIRECT : First, I'm not only checking the data has been received correctly, but can be retrieved properly from the storage medium. Without O_DIRECT I'm pretty much guaranteed that I'm just getting data from the page cache. To achieve the same without O_DIRECT , I had to use non-portable calls like sync_file_range on Linux to get the data to disk, then flush it from the page cache via madvise and finally reading it back. (Since I once learned the hard way that calling madvise with MADV_DONTNEED on a dirty page is essentially a noop).
But if anyone has a more elegant suggestion for this I'm all ears. :-)

Answer 1

I'm not only checking the data has been received correctly, but can be retrieved properly from the storage medium.

In order to do so, you need to defeat caching done by the device .

Most device data caches are relatively small - megabytes not gigabytes.[*] Provided your large files are sufficiently large, you can do write()+fsync() and you will overflow the device cache. Then use your DONTNEED to drop it from the Linux page cache. Then you can re-read() the file from the device.

[*] Some virtualized environments break this expectation. https://unix.stackexchange.com/questions/420299/why-is-sync-drop-caches-not-dropping-caches/