Process stdout different in console vs when redirected

Question

From a Windows C# application we want to run cipher.exe to remove data from unused disk space, in this case the D: drive:

cipher.exe /w:D:\

When done from a Windows command line, the output would be:

To remove as much data as possible, please close all other applications while
running CIPHER /W.
Writing 0x00
...................................................................................................
Writing 0xFF
...................................................................................................
Writing Random Numbers
...................................................................................................

Those lines with dots fill up incrementally over the duration of the cipher procedure. We thought we'd read those dots from a C# application to track the progress and display it on a progress bar. However we noticed that when the stdout is captured or redirected then the order is different:

cipher.exe /w:D:\ > out.txt

Results in a file with the following contents:

To remove as much data as possible, please close all other applications while
running CIPHER /W.
Writing 0x00
Writing 0xFF
Writing Random Numbers
...................................................................................................
...................................................................................................
...................................................................................................

And so when we try to capture that from a C# application, we don't read the dots until the very end of the process. For example when using the following code:

private void RunCipher()
{
    using (Process cipherProcess = new Process())
    {
        try
        {
            // Cipher does three phases, one with 00s, one with FFs and one with random bits
            // We count dots in the output for each phase to track the progress
            // The amount of dots per phase is always 99 (independent of the volume size)
            // See the end of this file to find the expected output

            cipherProcess.StartInfo.FileName = "cipher.exe";
            cipherProcess.StartInfo.Arguments = @"/w:D:\";

            cipherProcess.StartInfo.RedirectStandardOutput = true;
            cipherProcess.StartInfo.RedirectStandardError = true;
            cipherProcess.StartInfo.RedirectStandardInput = true;
            cipherProcess.StartInfo.UseShellExecute = false;
            cipherProcess.StartInfo.CreateNoWindow = true;

            cipherProcess.OutputDataReceived += CipherProcess_OutputDataReceived;
            cipherProcess.ErrorDataReceived += CipherProcess_ErrorDataReceived;
            cipherProcess.Exited += CipherProcess_Exited;

            cipherProcess.Start();

            cipherProcess.BeginOutputReadLine();
            cipherProcess.BeginErrorReadLine();
            cipherProcess.WaitForExit();
        }
        catch
        {
            Console.WriteLine("Exception occured");
        }
    }
    Console.WriteLine("Fininshed");
}

private void CipherProcess_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
    Console.WriteLine("OutputDataReceived: " + e.Data);
}

private void CipherProcess_Exited(object sender, EventArgs e)
{
    Console.WriteLine("Exited");
}

private void CipherProcess_ErrorDataReceived(object sender, DataReceivedEventArgs e)
{
    Console.WriteLine("ErrorDataReceived: " + e.Data);
}

The output of that is:

OutputDataReceived: To remove as much data as possible, please close all other applications while
OutputDataReceived: running CIPHER /W.
OutputDataReceived: Writing 0x00
The thread 0x41ac has exited with code 0 (0x0).
The thread 0x408c has exited with code 0 (0x0).
OutputDataReceived: Writing 0xFF
The thread 0x39e4 has exited with code 0 (0x0).
The thread 0x3e30 has exited with code 0 (0x0).
OutputDataReceived: Writing Random Numbers
The thread 0x34ac has exited with code 0 (0x0).
The thread 0x3960 has exited with code 0 (0x0).
OutputDataReceived: ...................................................................................................
OutputDataReceived: ...................................................................................................
OutputDataReceived: ...................................................................................................
ErrorDataReceived: 
OutputDataReceived: 
Fininshed

We also tried without OutputDataReceived+BeginOutputReadLine and instead using process.StandardOutput.Read() but it has the same problem: it first reads all three of the 'Writing (..)' output:

private void RunCipher2()
{
    using (Process cipherProcess = new Process())
    {
        try
        {
            cipherProcess.StartInfo.FileName = "cipher.exe";
            cipherProcess.StartInfo.Arguments = @"/w:D:\";

            cipherProcess.StartInfo.RedirectStandardOutput = true;
            cipherProcess.StartInfo.RedirectStandardError = true;
            cipherProcess.StartInfo.RedirectStandardInput = true;
            cipherProcess.StartInfo.UseShellExecute = false;
            cipherProcess.StartInfo.CreateNoWindow = true;

            cipherProcess.Start();

            while (!cipherProcess.StandardOutput.EndOfStream)
            {
                char nextChar = (char)cipherProcess.StandardOutput.Read();
                Console.Write(nextChar);
            }

            cipherProcess.WaitForExit();
        }
        catch
        {
            Console.WriteLine("Exception occured");
        }
    }
    Console.WriteLine("Fininshed");
}

And still the output is in order we wouldn't expect:

To remove as much data as possible, please close all other applications while
running CIPHER /W.
Writing 0x00
Writing 0xFF
Writing Random Numbers
...................................................................................................
...................................................................................................
...................................................................................................
Fininshed

We already found this thread, but that solution requires the redirected output to first work: Read Process StandardOutput before New Line Received

What's happening here? Is there a way to make this work? Or to track the progress in another way? Of course we could detect on the 'Writing' messages and report progress in thirds... but it seems like this should be possible :)

Answer 1

Assuming that the "..." comes out incrementally as progress is made, what you need to do is instead of using the DataRecieved event, is capture the StandardOutput stream and actually read it one character at a time. This way you'll see each individual full stop as it's written out. You can then count how many you've had and use that to infer the progress.