我在这里做错了什么...我的每个循环都很慢

Question

I am trying to concatenate the strings in two files and save it in a third file. But when the records from the first two files are more (say 100000+ records) my output file takes a long time to generate. What am i doing wrong here... Can someone please help

 var fileA = File.ReadAllLines("File1.txt");
 var fileB = File.ReadAllLines("File2.txt"); 

Then Do a cartesian of the Rows in the Files NxM where N and M represent the Number of rows in File1 and File2. So if there are 100 and 50 records each in File 1 and File2 Respectively, then the output is 100*50=5000

        FileStream fs = new FileStream("OutputFile.txt", FileMode.Create);
        // First, save the standard output.
        TextWriter tmp = Console.Out;
        StreamWriter sw = new StreamWriter(fs);


        foreach (var lst in cartesian)
        {
            Console.WriteLine(lst);
            Console.SetOut(sw);
            Console.WriteLine(lst);
            Console.SetOut(tmp);
            Console.WriteLine(lst);
        }

        sw.Close();

Answer 1

I don't think you're doing anything wrong. It just legitimately takes a long time to do a cartesian join of 100,000 x 100,000 records. You might improve performance a little bit by doing the join with nested for loops instead of LINQ, but your process is probably I/O bound.

Note that you don't need to use Console.SetOut , you can call WriteLine directly on sw :

foreach (var lst in cartesian)
{
  Console.WriteLine(lst);
  sw.WriteLine(lst);
  // and if you want to do it again: Console.WriteLine(lst);
}

Answer 2

Console.WriteLine() when writing to stdout is relatively heavy. See this test where I first just output 100000 lines to a text file with zero other processing, then the second test I write to stdout twice and call SetOut once each iteration. This is slightly different as your test writes to stdout twice but calls SetOut twice every iteration instead of only once.

FileStream fs = new FileStream(@"c:\temp\OutputFile.txt", FileMode.Create);
StreamWriter sw = new StreamWriter(fs);
TextWriter tmp = Console.Out; // stdout since it hasn't been changed
Console.SetOut(sw); // point to file
var stopw = Stopwatch.StartNew();
for (int i = 0; i < 100000; i++)
{               
    Console.WriteLine(i); // writes to file
}
sw.Dispose();
fs.Dispose();
var toFileTotalMs = stopw.Elapsed.TotalMilliseconds;

// Reset console to write to stdout
Console.SetOut(tmp);
stopw.Restart();
for (int i = 0; i < 100000; i++)
{
    Console.WriteLine(i); // writes to stdout
    Console.SetOut(tmp); // point to stdout (every iteration)
    Console.WriteLine(i); // writes to stdout
}
var toConsoleTotalMs = stopw.Elapsed.TotalMilliseconds;

Console.WriteLine($"toFileTotalMs={toFileTotalMs}; toConsoleTotalMs={toConsoleTotalMs};");

Console.Read(); // leaves console window open

Outputs:

toFileTotalMs = 17.7198 toConsoleTotalMs = 15964.9133

So it takes 900 times longer to do two Console.WriteLine() 's to stdout and call SetOut than it does to just write to the file. I just tried changing the original for loop to call SetOut every iteration in addition to writing to file and it went from 17.7ms to 43.8ms.