TextWriter Object not writing full document

Question

Hi I have a huge document which I have to read line by line using C# in .NET. Then perform a operation, and then write that line again.

I am testing the code with a smaller file, the actual file contains 992.482 lines. I have tried the following code to test:

while (!scenFile.EndOfStream)
{   writer.WriteLine(scenFile.ReadLine().ToString();
}

And I could only write 992.474. Then i tried using writer.Flush();

System.IO.TextWriter writer = File.CreateText(filepath);
StreamReader scenFile = new StreamReader(filepath2);

while (!scenFile.EndOfStream)
{   (here will go my do-something-function)
         {
          blah blah
         }
    writer.WriteLine(scenFile.ReadLine().ToString();
    writer.Flush();
}
writer.Close();

Then, I got all the lines. After inserting this line in the code, I checked, that the only way in that I could obtain is by typing "writer.Flush();" in every iteration. I have tried to insert it in a loop so that I use "writer.Flush();" every certain number of iterations, I have tried numbers from 50 to 500.000, and I can't get all the lines.

The problem is that I will have to perform the operation with a file which is 30 times the actual file, and I need to do it as fast as possible. Does anyone knows why is this and if there is any solution?

Thanks in advance

Answer 1

You don't need to to flush the stream twice. You should be able to just flush it before the close...

while (!scenFile.EndOfStream)
{   (here will go my do-something-function)
 {
  blah blah
 }
 writer.WriteLine(scenFile.ReadLine().ToString();
}
writer.Flush();
writer.Close();

Are you saying that somehow this didn't work?

Answer 2

Solved.

In C# Flush() won't only free the memory reserved for the buffer. It will also "writes it to the underlying stream" ( StreamWriter.Flush() ). Therefore what I did was just calling $writer.Flush() right after the $while .

This is my final solution will be

    System.IO.TextWriter writer = File.CreateText(filepath);
    StreamReader scenFile = new StreamReader(filepath2);
    int count = 0;
    while (!scenFile.EndOfStream)
    {   (here will go my do-something-function)
     {
      blah blah
     }
     writer.WriteLine(scenFile.ReadLine().ToString();
     count ++;
     if(count == 500000)
     {
         writer.Flush();
         count = 0;
     }
    }
   writer.Flush();
   writer.Close();