多个线程读取同一个文件

Question

I have a xml file that needs to be read from many many times. I am trying to use the Parallel.ForEach to speed this processes up since none of that data being read in is relevant as to what order it is being read in. The data is just being used to populate objects. My problem is even though I am opening the file each time in the thread as read only it complains that it is open by another program. (I don't have it opened in a text editor or anything :))

How can I accomplish multi reads from the same file?

EDIT: The file is ~18KB pretty small. It is read from about 1,800 times.

Thanks

Answer 1

If you want multiple threads to read from the same file, you need to specify FileShare.Read :

using (var stream = File.Open("theFile.xml", FileMode.Open, FileAccess.Read, FileShare.Read))
{
    ...
}

However, you will not achieve any speedup from this, for multiple reasons:

1. Your hard disk can only read one thing at a time. Although you have multiple threads running at the same time, these threads will all end up waiting for each other.
2. You cannot easily parse a part of an XML file. You will usually have to parse the entire XML file every time. Since you have multiple threads reading it all the time, it seems that you are not expecting the file to change. If that is the case, then why do you need to read it multiple times?

Answer 2

Depending on the size of the file and the type of reads you are doing it might be faster to load the file into memory first, and then provide access to it directly to your threads.

You didnt provide any specifics on the file, the reads, etc so I cant say for sure if it would address your specific needs.

The general premise would be to load the file once in a single thread, and then either directly (via the Xml structure) or indirectly (via XmlNodes, etc) provide access to the file to each of your threads. I envision something similar to:

1. Load the file
2. For each Xpath query dispatch the matching nodes to your threads.

If the threads dont modify the XML directly, this might be a viable alternative.

Answer 3

When you open the file, you need to specify FileShare.Read :

using (var stream = new FileStream("theFile.xml", FileMode.Open, FileAccess.Read, FileShare.Read))
{
    ...
}

That way the file can be opened multiple times for reading

Answer 4

While an old post, it seems to be a popular one so I thought I would add a solution that I have used to good effect for multi-threaded environments that need read access to a file. The file must however be small enough to hold in memory at least for the duration of your processing, and the file must only be read and not written to during the period of shared access.

string FileName = "TextFile.txt";
string[] FileContents = File.ReadAllLines(FileName);

foreach (string strOneLine in FileContents)
{
  // Do work on each line of the file here
}

So long as the file is only being read, multiple threads or programs can access and process it at the same time without treading on one another's toes.