C# increase the size of reading binary data

Question

I am using the below code from Jon Skeet's article . Of late, the binary data that needs to be processed has grown multi-fold. The binary data size file size that I am trying to import is ~ 900 mb almost 1 gb. How do I increase the memory stream size.

public static byte[] ReadFully (Stream stream)
{
    byte[] buffer = new byte[32768];
    using (MemoryStream ms = new MemoryStream())
    {
        while (true)
        {
            int read = stream.Read (buffer, 0, buffer.Length);
            if (read <= 0)
                return ms.ToArray();
            ms.Write (buffer, 0, read);
        }
    }
}

Answer 1

Your method returns a byte array, which means it will return all of the data in the file. Your entire file will be loaded into memory.

If that is what you want to do, then simply use the built in File methods:

byte[] bytes = System.IO.File.ReadAllBytes(string path);
string text = System.IO.File.ReadAllText(string path);

If you don't want to load the entire file into memory, take advantage of your Stream

using (var fs = new FileStream("path", FileMode.Open))
using (var reader = new StreamReader(fs))
{
    var line = reader.ReadLine();
    // do stuff with 'line' here, or use one of the other
    // StreamReader methods.
}

Answer 2

You don't have to increase the size of MemoryStream - by default it expands to fit the contents.

Apparently there can be problems with memory fragmentation , but you can pre-allocate memory to avoid them:

using (MemoryStream ms = new MemoryStream(1024 * 1024 * 1024))  // initial capacity 1GB
{
}

In my opinion 1GB should be no big deal these days, but it's probably better to process the data in chunks if possible. That is what Streams are designed for.