读取多个非常大的文件的最佳方法

Question

I need help figuring out the fastest way to read through about 80 files with over 500,000 lines in each file, and write to one master file with each input file's line as a column in the master. The master file must be written to a text editor like notepad and not a Microsoft product because they can't handle the number of lines.

For example, the master file should look something like this:

File1_Row1,File2_Row1,File3_Row1,...

File1_Row2,File2_Row2,File3_Row2,...

File1_Row3,File2_Row3,File3_Row3,...

etc.

I've tried 2 solutions so far:

1. Create a jagged array to hold each files' contents into an array and then once reading all lines in all files, write the master file. The issue with this solution is that Windows OS memory throws an error that too much virtual memory is being used.
2. Dynamically create a reader thread for each of the 80 files that reads a specific line number, and once all threads finish reading a line, combine those values and write to file, and repeat for each line in all files. The issue with this solution is that it is very very slow.

Does anybody have a better solution for reading so many large files in a fast way?

Answer 1

The best way is going to be to open the input files with a StreamReader for each one and a StreamWriter for the output file. Then you loop through each reader and read a single line and write it to the master file. This way you are only loading one line at a time so there should be minimal memory pressure. I was able to copy 80 ~500,000 line files in 37 seconds. An example:

using System;
using System.Collections.Generic;
using System.IO;
using System.Diagnostics;

class MainClass
{
    static string[] fileNames = Enumerable.Range(1, 80).Select(i => string.Format("file{0}.txt", i)).ToArray();

    public static void Main(string[] args)
    {
        var stopwatch = Stopwatch.StartNew();
        List<StreamReader> readers = fileNames.Select(f => new StreamReader(f)).ToList();

        try
        {
            using (StreamWriter writer = new StreamWriter("master.txt"))
            {
                string line = null;
                do
                {
                    for(int i = 0; i < readers.Count; i++)
                    {
                        if ((line = readers[i].ReadLine()) != null)
                        {
                            writer.Write(line);
                        }
                        if (i < readers.Count - 1)
                            writer.Write(",");
                    }
                    writer.WriteLine();
                } while (line != null);
            }
        }
        finally
        {
            foreach(var reader in readers)
            {
                reader.Close();
            }
        }
        Console.WriteLine("Elapsed {0} ms", stopwatch.ElapsedMilliseconds);
    }
}

I've assume that all the input files have the same number of lines, but you should be add the logic to keep reading when at least one file has given you data.

Answer 2

Use Memory Mapped files seems what is suitable to you. Something that does not execute pressure on memory of your app contemporary maintaining good performance in IO operations.

Here complete documentation: Memory-Mapped Files

Answer 3

If you have enough memory on the computer, I would use the Parallel.Invoke construct and read each file into a pre-allocated array such as:

string[] file1lines = new string[some value];
string[] file2lines = new string[some value];
string[] file3lines = new string[some value];

Parallel.Invoke(
() =>
{
   ReadMyFile(file1,file1lines);
},
() =>
{
   ReadMyFile(file2,file2lines);
},
() =>
{
   ReadMyFile(file3,file3lines);
}
);

Each ReadMyFile method should just use the following sample code which, according to these benchmarks , is the fastest way to read a text file:

int x = 0;
using (StreamReader sr = File.OpenText(fileName))
{
        while ((file1lines[x] = sr.ReadLine()) != null)
        {
               x += 1;
        }
}

If you need to manipulate the data from each file before writing your final output, read this article on the fastest way to do that.

Then you just need one method to write the contents to each string[] to the output as you desire.

Answer 4

Have an array of open file handles. Loop through this array and read a line from each file into a string array. Then combine this array into the master file, append a newline at the end.

This differs from your second approach that it is single threaded and doesn't read a specific line but always the next one.

Of course you need to be error proof if there are files with less lines than others.