多线程读取随机文件

Question

i have a thread pool and all threads have a fixed set of images. Let's say they have each 100 images. I need to detect which image contains another image that reside in an image bank.

• thread 1 - 100 images

• thread 2 - 100 images

• thread 3 - 100 images

• thread 4 - 100 images

• Image Base - 50 images

Now i need all the threads to look inside the Image base to see if one of the images they hold resembles one of the image base. I have my image matching done, what i am worried about is if multiple threads might open the same image file. What would be the proper way of tackling this ? It would be nice not to "lock" all the other threads for each IO.

Thanks !

Answer 1

What about something like this, where each thread has a reference to the image bank and provides a delegate that will be called for each of the files in the image bank? Here is a skeleton of what the image bank might look like.

public class ImageBank {
    public delegate bool ImageProcessorDelegate(String imageName);

    private readonly List<String> _imageBank;

    public ImageBank() 
    {
        // initialize _imageBank with list of image file names
    }

    private int ImageBankCount {
        get {
            lock(_imageBank) {
                return _imageBank.Count;
            }
        }   
    }

    private List<String> FilterImageBank(ISet<String> exclude) 
    {
        lock(_imageBank)
        {
            return _imageBank.Where(name => !exclude.Contains(name)).ToList();
        }
    }

    public void ProcessImages(ImageProcessorDelegate imageProcessor)
    {
        var continueProcessing = true;
        var processedImages = new HashSet<String>();
        var remainingImages = new List<String>();

        do
        {
            remainingImages = FilterImageBank(processedImages);

            while(continueProcessing && remainingImages.Any())
            {
                var currentImageName = remainingImages[0];
                remainingImages.RemoveAt(0);

                // protect this image being accessed by multiple threads.
                var mutex = new Mutex(false, currentImageName);             
                if (mutex.WaitOne(0))
                {
                    try 
                    {
                        // break out of loop of the processor found what it was looking for.
                        continueProcessing = imageProcessor(currentImageName);
                    }
                    catch (Exception)
                    {
                        // exception thrown by the imageProcessor... do something about it.
                    }
                    finally
                    {
                        // add the current name to the set of images we've alread seen and reset the mutex we acquired
                        processedImages.Add(currentImageName);
                        mutex.ReleaseMutex();
                    }
                }
            }
        }
        while(continueProcessing);
    }
}

Then, each thread would have its list of images ( _myImageList ) and a ThreadProc that looks something like this:

void ThreadProc(object bank) 
{
  var imageBank = bank as ImageBank;
  foreach(var myImage in _myImageList)
  {
    imageBank.ProcessImages(imageName => 
      {
         // do something with imageName and myImage
         // return true to continue with the next image from the image bank
         // or false to stop processing more images from the image bank
      }
    );
  }
}

Answer 2

Assuming all threads have the same set of images they have to work on, and assuming the pahts to these files are in a list or some other collection, you could try something like this:

// A shared collection
List<string> paths = new List<string>();
// Fill this collection with your fixed set.

IEnumerator<T> e = paths.GetEnumerator();
// Now create all threads and use e as the parameter. Now all threads have the same enumerator.

// Inside each thread you can do this:
while(true)
{    
    string path;
    lock(e)
    {
        if (!e.MoveNext())
            return; // Exit the thread.
        path = e.Current;
    }
    // From here, open the file, read the image, process it, etc.
}

In this example you only put a lock on the enumerator. Only one thread can read from it at the same time. So each time it is called, a different path will come out of it.

Outside the lock you can do all the processing, I/O, etc.

Of course the collection could also be of another type, like an array.