Concurrency reading writing files

Question

Currently I am writing the windows service to process massive xml files and save info from xml file to database. There are 3 machines running the same services that point to the same share location. I always got exception that File Not Found during the process and don't know how to solve this problems. I tried to get all file from the directory and rename those files based on the machine name and allow each machine to process it own set of files but still suffer from File Not Found exception. Could anyone show me the proper way to deal with this.

Thank you.

Code

if (Directory.Exists(folder))
{
  string pattern = ".xml";
  string machineName = System.Environment.MachineName;
  string[] files = Directory.GetFiles(folder, pattern, SearchOption.AllDirectories);  
  newExt = string.Format("{0}.{1}", machineName, newExt);
  for (int i = 0; i < files.Length; i++)
  {
      if (files[i].Contains(machineName))
      {
           //replace this new extension
           files[i].Replace(machineName + ".", "");
      }
      else
      {
           files[i] = ChangeExtension(files[i], newExt, true);
      }
  }
  IEnumerable<string> sortedFiles = files.Where(f => !string.IsNullOrEmpty(f) && f.Contains(machineName))
                                         .OrderBy(f => f, Sorter);
}

Answer 1

It's hard to see what's happening without looking at your code, but I'll speculate that the three individual servers get a directory listing, select a file from that listing to work on, and in the mean time a different server has renamed the file because the other server decided to start working on the file.

If this happens infrequently, you can ignore the Exception (assuming my explanation is correct). If this happens frequently you will need to explain in more detail what you are doing.

You can minimize contention for the same file by having each server sort the file listing in a different way (eg one AZ, the second ZA, the third LZ, AJ, ...) and then pick the top one in the sort order to work on.

Answer 2

Could you, please, explain with what names files appear in the directory?

1. If files already have the target machinename in their names it is enough for each machine to process only those named files (or not??).
2. If files appear without any special machinename specified, it looks like a race among the working services, who captures a file first. Then your exception seems to be expectable (just as Eric J. said), and you'd just try..catch it and omit the file. It is normall and some other service has already captured that file. Or you could think over some resource sharing technique to avoid several services to check the directory at once time. At least you could make some subdirectories to divide all files into some groups and make each service work with each subdirectory...

PS: surely, if you feel, that I failed to catch the issue, could you, please, provide some more accurate info.

Answer 3

As Eric J mentioned, you could simply ignore the errors:

if (Directory.Exists(folder))
{
  string pattern = ".xml";
  string machineName = System.Environment.MachineName;
  string[] files = Directory.GetFiles(folder, pattern, SearchOption.AllDirectories);  
  newExt = string.Format("{0}.{1}", machineName, newExt);

  for (int i = 0; i < files.Length; i++)
  {
      try
      {
          if (files[i].Contains(machineName))
          {
               //replace this new extension
               files[i].Replace(machineName + ".", "");
          }
          else
          {
               files[i] = ChangeExtension(files[i], newExt, true);
          }
     catch(FileNotFoundException ex)
     {   
     } 
  }

  IEnumerable<string> sortedFiles = files.Where(f => !string.IsNullOrEmpty(f) && f.Contains(machineName))
                                         .OrderBy(f => f, Sorter);
}