如何在C＃中使用线程读取和解析非常大的平面文件？

Question

I have to read a large text file and to parse it line by line using C#. It could be done easily with StreamReader for small sized file but it caught out of memory exception while working with large file. How can I adapt it for large files?

Following code catches OutOfMemoryException :

using (StreamReader reader = new StreamReader(FileNameWithPath))
{
    while ((line = reader.ReadLine()) != null)
    {
        // Do something here...
    }
}

Answer 1

That is pretty much the standard code for a lazy line reader, and shouldn't cause an OutOfMemoryException unless there are some really big single lines. You could also try:

foreach(var line in File.ReadLines(FileNameWithPath)) {
    // Do something here...
}

which just makes it cleaner, but does the same thing. So there are two options:

1. one or more of the "lines" is simply huge
2. something in "Do something here" is slowly (or quickly) eating your memory

I expect the latter is more likley.

Answer 2

I am not sure with this but give try to this class of .net framework

MemoryMappedFile Class -A memory-mapped file maps the contents of a file to an application's logical address space. Memory-mapped files enable programmers to work with extremely large files because memory can be managed concurrently, and they allow complete, random access to a file without the need for seeking. Memory-mapped files can also be shared across multiple processes.

Answer 3

using (var inputFile = new System.IO.StreamReader(sourceFilePath))
{
    while (inputFile.Peek() >= 0) {
        string lineData = inputFile.ReadLine();

        // Do something with lineData
    }
}

Answer 4

How about specify the buffer size ?

like this.

using (var reader = new StreamWriter(path,false,Encoding.UTF8, 1000))
{
    .....

}