使用代理进行异步调用

Question

I want that separate Async threads of method splitFile should run so that the task will become faster but below code is not working. When I debug , it reaches till line RecCnt = File.ReadAllLines(SourceFile).Length - 1; and comes out. Please help.

public delegate void SplitFile_Delegate(FileInfo file);

static void Main(string[] args)
{ 
     DirectoryInfo d = new DirectoryInfo(@"D:\test\Perf testing Splitter"); //Assuming Test is your Folder
     FileInfo[] Files = d.GetFiles("*.txt"); //Getting Text files

     foreach (FileInfo file in Files)
     {
         SplitFile_Delegate LocalDelegate = new SplitFile_Delegate(SplitFile);

         IAsyncResult R = LocalDelegate.BeginInvoke(file, null, null); //invoking the method
         LocalDelegate.EndInvoke(R);

      }
}

private static void SplitFile(FileInfo file)
{
     try
     {
         String fname;
         //int FileLength;
         int RecCnt;
         int fileCount;
         fname = file.Name;
         String SourceFile = @"D:\test\Perf testing Splitter\" + file.Name;


         RecCnt = File.ReadAllLines(SourceFile).Length - 1;
         fileCount = RecCnt / 10000;

         FileStream fs = new FileStream(SourceFile, FileMode.Open);
         using (StreamReader sr = new StreamReader(fs))
         {
             while (!sr.EndOfStream)
             {
                 String dataLine = sr.ReadLine();
                 for (int x = 0; x < (fileCount + 1); x++)
                 {
                      String Filename = @"D:\test\Perf testing Splitter\Destination Files\" + fname + "_" + x + "by" + (fileCount + 1) + ".txt"; //test0by4
                      using (StreamWriter Writer = file.AppendText(Filename))
                      {
                          for (int y = 0; y < 10000; y++)
                          {
                              Writer.WriteLine(dataLine);
                              dataLine = sr.ReadLine();
                          }
                          Writer.Close();
                      }

                  }
              }
          }
     }
     catch (Exception ex)
     {
          Console.WriteLine(ex.Message);
     }
}

Answer 1

Your code doesn't really need any multi-threading. It doesn't really even need asynchronous processing all that much - you're saturating the I/O most likely, and unless you've got multiple drives as the data sources, you're not going to improve that by adding parallelism.

On the other hand, your code is reading each file twice . For no reason, wasting memory, time and even CPU. Instead, just do this:

FileStream fs = new FileStream(SourceFile, FileMode.Open);
using (StreamReader sr = new StreamReader(fs))
{
    string line;
    string fileName = null; 
    StreamWriter outputFile = null;
    int lineCounter = 0;
    int outputFileIndex = 0;

    while ((line = sr.ReadLine()) != null)
    {
        if (fileName == null || lineCounter >= 10000)
        {
            lineCounter = 0;
            outputFileIndex++;
            fileName = @"D:\Output\" + fname + "_" + outputFileIndex + ".txt";

            if (outputFile != null) outputFile.Dispose();
            outputFile = File.AppendText(fileName);
        }

        outputFile.WriteLine(line);
        lineCounter++;
    }
}

If you really need to have the filename in format XOutOfY , you can just rename them afterwards - it's a lot cheaper than reading the source file twice, line after line. Or, if you don't care about keeping the whole file in memory at once, just use the array you got from ReadAllLines and iterate over that, rather than doing the reading all over again.

To make this even easier, you can also use foreach (var line in File.ReadLines(fileName)) .

If you really want to make this asynchronous, the way to handle that is by using asynchronous I/O, not just by spooling new threads. So you can use await with StreamReader.ReadLineAsync etc.

Answer 2

You are not required to call EndInvoke and really all EndInvoke does is wait on the return value for you. Since SplitFile returns void, my guess is there's an optimization that kicks in because you don't need to wait on anything and it simply ignores the wait. For more details: C# Asynchronous call without EndInvoke?

That being said, your usage of Begin/EndInvoke will likely not be faster than serial programming (and will likely be marginally slower) as your for loop is still serialized, and you're still running the iteration in serial. All that has changed is you're using a delegate where it looks like one isn't necessary.

It's possible that what you meant to use was Parallel.ForEach (MSDN: https://msdn.microsoft.com/en-us/library/dd992001(v=vs.110).aspx ) which will potentially run iterations in parallel.

Edit: As someone else has mentioned, having multiple threads engage in file operations will likely not improve performance as your file ops are probably disk bound. The main benefit you would get from an async file read/write would probably be unblocking the main thread for a UI update. You will need to specify what you want with "performance" if you want a better answer.