data integrity issue with ntfs but not ext3

Question

In my application I am continually writing data to file1 and flushing it to the device. In another thread, I am reading data from file1 and writing it to file2 .

Every time I do the fwrite + fflush on file1, I signal to the other thread to start reading from it. The other thread reads data from file1 and dumps it into file2. Pretty simple logic. Additionally, after every few minutes, I seek back to start of file1 and start overwriting old data.

Now my problem is that once I start overwriting data in file1 , the data read into file2 is sometimes the old data (ie data written in the previous iteration) even though writer thread has signaled that it wrote the new data (and flushed it).

I am writing to and reading from a solid state drive (128 GB SAMSUNG 470 Series, if that helps) on [C + linux + arm platfrom]. I feel that there is an issue with the processor cache. Perhaps the write goes into the cache and the read by the reader thread comes from the flash, and hence the stale data.

The catch here is that this problem occurs if the SSD is formatted with NTFS. If I format it with ext3, the problem goes away. Unfortunately, NTFS is a hard requirement. Another interesting observation is that if I have two reader threads, both get stale data at different instants.

Event after disabling the SSD write cache (with hdparm -W0 /dev/sda1 ), I get the same problem with NTFS. I am badly stuck up with this since more than a week.

Any idea what is happening, and why is it happening that way?

Any help will be worth its weight in gold...

EDIT Turns out that the NTFS driver does not like me overwriting a file by rewinding the file pointer. Is this a known thing?

Answer 1

Ok, so I found the issue myself (and how rarely does that happen.!!).

I found that there was a problem with the C library buffering (fread/fwrite). So I do fflush() before every fread(). This solves my problem (I don't know what exactly went wrong with the driver but I am assuming that there is some issue with the "read" buffering of the C library I/O functions, when reading from the same location of the file second time around).

Thanks @Asad Rasheed and @jrtipton for your inputs:)