读取和写入特定行到文本文件 C#

Question

I have a master file called FileName with IDs of people. It is in sorted order. I want to divide IDs into 27 chunks and copy each chunk into a different text file.

using (FileStream fs = File.Open(FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
         {
            string line;
      int numOfLines = File.ReadAllLines(FileName).Length;  -- I have 73467
       int eachSubSet = (numOfLines / 27);
       var lines = File.ReadAllLines(dataFileName).Take(eachSubSet);
       File.WriteAllLines(FileName1,lines);
      }

I have 27 different text files. so I want 73467 of IDs divided equally and copied over to 27 different files. So, 1st file will have ID#1 to ID#2721 2nd Dile will have ID#2722 to ID#(2722+2721) and so on. I do not know how to automate this and run this quickly.

Thanks HR

Answer 1

The simplest way would be to run File.ReadLine and WriteLine inside a loop and decide what file will receive which line.

I wouldn't recommend you to parallelize this routine since it's an IO operation, but just the copy of lines would be pretty fast.

Answer 2

Note that in your sample code you called File.ReadAllLines twice, so you actually parse your entire input file twice. So avoiding that should speed up the process, and also you didn't actually split the files, you only wrote the first file out of the 27. Untested, but something along these lines should work:

const int numOfFiles = 27;
string[] lines = File.ReadAllLines(FileName);
int numOfLines = lines.Length;
int eachSubSet = numOfLines/numOfFiles;
int firstSubset = numOfLines%numOfFiles + eachSubSet;
IEnumerable<string> linesLeftToWrite = lines;
for (int index = 0; index < numOfFiles; index++)
{
    int numToTake = index == 0 ? firstSubset : eachSubSet;
    File.WriteAllLines(string.Format("{0}_{1}.txt", FileName, index), linesLeftToWrite.Take(numToTake));
    linesLeftToWrite = linesLeftToWrite.Skip(numToTake);
}