使用一个内存流进行多次文件搜索迭代

Question

I'm having several methods which each apply an operation to the a textfile, where the next operation requires the result of the previous operation as input:

private TextReader input = new StreamReader("input.txt");
private TextWriter output = new StreamWriter("output.txt");
MemoryStream result_1 = new MemoryStream();
MemoryStream result_2 = new MemoryStream();

Operation_1(input, ref result_1);
Operation_2(result_1, ref result_2);
Operation_3(result_2, output);

The code for Operation_1:

   private void Operation_1(TextReader input, ref MemoryStream output)
    {
        TextWriter outputWriter = new StreamWriter(output);
        String line;

        while (input.Peek() >= 0) //while not end of file
        {
            line = input.ReadLine();
            //perform operation on line
            outputWriter.writeline(line);
        }
        input.Close();
    }

the code for operation_2:

   private void Operation_2(TextReader input, ref MemoryStream output)
    {
        input.Seek(0, SeekOrigin.Begin); //reset stream to start of file
        TextReader inputReader = new StreamReader(input);
        TextWriter outputWriter = new StreamWriter(output);
        String line;

        while (inputReader.Peek() >= 0) //while not end of file
        {
            line = inputReader.ReadLine();
            //perform operation on line
            outputWriter.writeline(line);
        }
        inputReader.Close();
    }

The code for operation_3:

    private void operation_3(MemoryStream input, TextWriter output)
    {
       input.Seek(0, SeekOrigin.Begin);  //reset stream to start of file
       TextReader inputReader = new StreamReader(input);
       String line;

       while (inputReader.Peek() >= 0) //while not end of file
       {
            line = inputReader.ReadLine();
            //perform operation on line
            output.writeline(line);
       }
       inputReader.Close();
       output.Close();
    }

Now the problem is that i'm not getting the same result as storing each intermediate result to a physical txt file on the harddisk and using that file for the next operation. A few lines and the end of the file is missing.

Also this seems like not a very clean and generic way of doing it.

So hence my question; why are my results different when using MemoryStream for the intermediate results and is there a cleaner, more flexible way of doing this? (I want to work towards a solution were it is possible to choose if you want to save the intermediate results or not).

Answer 1

• They are different, because you forgot to flush your writers.
• The ref modifiers are not needed, because you don't create a new MemoryStream in your operations

Your methods would be a bit cleaner this way:

private void Operation_1(TextReader input, Stream output)
{
    TextWriter outputWriter = new StreamWriter(output);
    String line;

    outputWriter.Write(input.ReadToEnd());
    outputWriter.Flush();

    input.Close();
}

private void Operation_2(Stream input, Stream output)
{
    input.Seek(0, SeekOrigin.Begin); //reset stream to start of file
    TextReader inputReader = new StreamReader(input);
    TextWriter outputWriter = new StreamWriter(output);

    outputWriter.Write(inputReader.ReadToEnd());
    outputWriter.Flush();
    inputReader.Close();
}

The code for operation_3:

private void operation_3(Stream input, TextWriter output)
{
   input.Seek(0, SeekOrigin.Begin);  //reset stream to start of file
   TextReader inputReader = new StreamReader(input);

   output.Write(inputReader.ReadToEnd());

   inputReader.Close();
   output.Flush();
   output.Close();
}

Answer 2

I have a suggestion to change your code. Maybe instead of using Streams and TextReaders you could just work with IEnumerable.

Please see the sample below (It's just a sample. Error handling not included to keep the sample simple.)

The result of the previous operation is provided as parameter to the next. Therefore, it get's executed as following Operation3(Operation2(Operation1))).

The first sample below is reading and changing the lines contained in the file line by line The second sample below is reading the whole file and changing the lines providing all lines to the next operation (lines.ToArray() reads the whole file).

With streams you always have to be careful that they are disposed correctly and at the desired time (for example StreamReader closes the inner stream by default when StreamReader gets disposed).

using System;
using System.Collections.Generic;
using System.IO;
using System.Globalization;
using System.Linq;

namespace ConsoleApplication2
{
    class Program
    {
        static void Main()
        {
            //line per line...
            File.WriteAllLines
                (
                    @"C:\temp\output.txt",
                    ChangeLines(File.ReadLines(@"C:\temp\input.txt"),
                                line =>
                                LineOperation3
                                    (
                                        LineOperation2
                                            (
                                                LineOperation1(line)
                                            )
                                    )
                        )
                );

            //lines per lines...
            File.WriteAllLines
               (
                   @"C:\temp\output2WithCount.txt",
                   ChangeLines(File.ReadLines(@"C:\temp\input.txt"),
                               lines =>
                                        LinesCountOperation
                                        (
                                            LinesCountOperation
                                            (
                                                LinesCountOperation(lines,LineOperation1),
                                                LineOperation2
                                            )
                                            , LineOperation3
                                          )
                       )
               );
        }

        private static IEnumerable<string> ChangeLines(IEnumerable<string> lines, Func<string, string> lineFunc)
        {
            foreach (var line in lines)
            {
                yield return lineFunc(line);
            }
        }

        private static IEnumerable<string> ChangeLines(IEnumerable<string> lines, Func<IEnumerable<string>, IEnumerable<string>> linesFunc)
        {
            foreach(var changedLine in linesFunc(lines))
            {
                if (changedLine != null)
                {
                    yield return changedLine;
                }
            }
        }

        private static IEnumerable<string> LinesCountOperation(IEnumerable<string> lines, Func<string, string> lineFunc)
        {
            var readAllLines = lines.ToArray();
            var linesCount = readAllLines.Count();

            foreach (var line in readAllLines)
            {
                var changedLine = lineFunc(line);
                if (changedLine == null)
                {
                    continue;
                }
                yield return string.Format(CultureInfo.InvariantCulture, "{0}-{1}", linesCount, changedLine);
            }
        }

        private static string LineOperation1(string line)
        {
            return string.Format(CultureInfo.InvariantCulture, "{0}{1}", line, "1");
        }

        private static string LineOperation2(string line)
        {
            return string.Format(CultureInfo.InvariantCulture, "{0}{1}", line, "2");
        }

        private static string LineOperation3(string line)
        {
            return string.Format(CultureInfo.InvariantCulture, "{0}{1}", line, "3");
        }
    }
}