从C＃中的巨大MemoryStream中读取

Question

I use a BinaryReader ( MemoryStream(MyByteArray) ) to read variable sized records and process them all in memory. This works well as long as my bytestream, which is in the array, is less than about 1.7 GB in size. After that (which is the maximum size of an integer in my 64-bit system) you cannot create a larger bytearray, although I have enough real memory. So my solution has been to read the bytestream and split it into several byte arrays.

Now however, I cannot "read" across the byte array boundaries, and, as my data is in a variable format, I cannot ensure that byte arrays always finish on a whole record.

This must be a common problem for people processing very large datasets and still have the need for speed.

How do I handle this problem?

Answer 1

Edit : Reading up on the basics, I realize that memory-mapped files might be slower than normal I/O for sequential access.

Have you tried something like this:

var stream = new FileStream("data", 
    FileMode.Open, 
    FileAccess.Read, 
    FileShare.Read, 
    16 * 1024, 
    FileOptions.SequentialScan)

var reader = new BinaryReader(stream);

If your data resides in a file and you can use .NET 4.0 consider using MemoryMappedFile .

You can then either use a MemoryMappedViewStream to get a stream or use a MemoryMappedViewAccessor to get a BinaryReader -like interface.

Answer 2

For excessively large streams, you shouldn't try dumping it in MemoryStream - use things like FileStream instead, and talk directly to disk. The inbuilt buffering is usually sufficient, or you can tweak this with things like BufferedStream (but I have rarely needed to - but then, I tend to include my own data-processing buffer).

You might also consider things like compression or densely packed data, and serializers designed to work by streaming records rather than creating an entire graph at once (although since you mention BinaryReader , you may already be doing this highly manually, so this might not be an issue).