使用C＃中的线程将大文本文件（500万条记录）并行拆分为较小的文件

Question

I have a large text file containing 5 million of records (5 columns and 5 million rows). The image of file is shown below

For splitting, I used the concept of threading. I created 10 threads for splitting the larger file. I have used the string array to store the values while reading the larger file. The code is shown below.

class Program
{
    const string sourceFileName = @"C:\Users\Public\TestFolder\ThreadingExp\NewMarketData.txt";
    const string destinationFileName = @"C:\Users\Public\TestFolder\ThreadingExp\NewMarketData-Part-{0}.txt";

    static void Main(string[] args)
    {
        int[] index = new int[20];
        index[0] = 0;
        for(int i=1;i<11;i++)
        {
            index[i] = index[i-1]+500000;
        }

        //Reading Part
        var sourceFile = new StreamReader(sourceFileName);
        string[] ListLines = new string[5000000];
        for (int i = 0; i < 5000000; i++)
        {
            ListLines[i] = sourceFile.ReadLine();
        }            

        //Creating array of threads
        Thread[] ArrayofThreads = new Thread[10];
        for (int i = 0; i < ArrayofThreads.Length; i++)
        {
            ArrayofThreads[i] = new Thread(() => Writing(ListLines,index[i], index[i+1]));
            ArrayofThreads[i].Start();
        }

        for (int i = 0; i < ArrayofThreads.Length; i++)
        {
            ArrayofThreads[i].Join();
        }
    }
    static void Writing(string[] array, int a, int b)
    {
        //Getting the thread number
        int id= Thread.CurrentThread.ManagedThreadId;

        var destinationFile = new StreamWriter(string.Format(destinationFileName,id));

        string line;
        for (int i = a; i< b;i++ )
        {
            line = array[i];
            destinationFile.WriteLine(line);
        }

        destinationFile.Close();         
    }

}   

The code works fine. Writing to different files is done in parallel here. But for reading, I have stored the whole content in a single array and then pass through different threads for writing using indexing. I want to do both the tasks (read the larger file and write in the different small files) in parallel using threads.

Answer 1

You're almost certainly better off doing this with a single thread.

First, you must read the text file sequentially. There's no shortcut that will let you skip ahead and find the 500,000th line without first reading the 499,999 lines that come before it.

Second, even if you could do that, the disk drive can only service a single request at a time. It can't be reading from two places at the same time. So while you're reading one part of the file, the thread that wants to read another part of the file is just sitting there waiting for the disk drive.

Finally, unless your output files are on separate drives, you have the same problem as with reading: the disk drive can only do one thing at a time.

So you're better off to start with something simple:

const int maxLinesPerFile = 5000000;
int fileNumber = 0;
var destinationFile = File.CreateText("outputFile"+fileNumber);

int linesRead = 0;
foreach (var line in File.ReadLines(inputFile))
{
    ++linesRead;
    if (linesRead > maxLinesPerFile)
    {
        destinationFile.Close();
        ++fileNumber;
        destinationFile = File.CreateText("outputFile"+fileNumber);
    }
    destinationFile.WriteLine(line);
}
destinationFile.Close();

If your input and output files are on separate drives, you could potentially save a little bit of time by having two threads: one for input and one for output. They would communicate using a BlockingCollection . Basically, the input thread would put lines onto the queue and the output thread would read from the queue and output the files. In theory that will overlap the reading time with the writing time, but the truth is that the queue fills up and the reader ends up having to wait on the writing thread. You get some increase in performance, but not nearly what you'd expect.