在将文件添加到zip文件中时，在C＃中使用BinaryWriter内存不足

Question

I am trying to add files to a Zip file, preserving the directory. The code below is basically working as long as I do not have files of a few 100 Mb to zip. If I just zip a directory with 1 file of about 250 Mb (on a system with plenty of memory BTW) I get an OutOfMemory exception on the write.Write() line.

I already modified the code to read in chunks as it first failed when I read/wrote the whole file. I don't know why it still fails?

    using (FileStream zipToOpen = new FileStream(cZipName, eFileMode)) 
        ZipArchiveEntry readmeEntry = archive.CreateEntry(cFileToBackup

);

    using (BinaryWriter writer = new BinaryWriter(readmeEntry.Open()))
    {
        FileStream fsData = null;                                                                // Load file into FileStream
        fsData = new FileStream(cFileFull, FileMode.Open, FileAccess.Read);
        {
            byte[] buffer = new byte[1024];
            int bytesRead = 0;
            while ((bytesRead = fsData.Read(buffer, 0, buffer.Length)) > 0)
            {
                 writer.Write(buffer,0,bytesRead); // here it fails
                 fsData.Flush(); // ->CHANGED  THIS TO writer.Flush() SOLVED IT - nearly..
            }
        }
        fsData.Close();
    }

EDIT : Arkadiusz K was right that I used the flush on the reader, not the writer. After changing that, the program zips files of 1 Gb or more where it stopped at 100 Mb first. However, I get another exception when I try to zip eg a 6 Gb file - it stops with: System.IO.IOException was unhandled Stream was too long Source=mscorlib StackTrace: at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) (etc)

Does anybody have an idea why it still fails? I'd say th code should now properly read and write 1 Kb at a time?

Answer 1

First of all, I'd really like to format your code and make it as succinct as it should be:

var readmeEntry = archive.CreateEntry(cFileToBackup);
using (var fsData = new FileStream(cFileFull, FileMode.Open, FileAccess.Read))
using (var writer = new BinaryWriter(readmeEntry.Open()))
{
    var buffer = new byte[1024];
    int bytesRead;
    while ((bytesRead = fsData.Read(buffer, 0, buffer.Length)) > 0)
    {
         writer.Write(buffer, 0, bytesRead); // here it fails
         writer.Flush();
    }
}

Now, to explain why it fails:

BinaryWriter is a stream writer. When it has to write the data to the stream, it usually writes it Length-prefixed and:

Length-prefixed means that this method first writes the length of the string, in bytes, when encoded with the BinaryWriter instance's current encoding to the stream. This value is written as an unsigned integer. This method then writes that many bytes to the stream.

In order to write to the file, in your case, the data is written to MemoryStream first. Here, MemoryStream is the backing store stream. Refer to the diagram below:

^{(Image taken from: http://kcshadow.net/wpdeveloper/sites/default/files/streamd3.png )}

Because, either your system's memory is about 6-8GB or because your application has been allocated that much memory only, the backing store stream is expanded to the max possible when you attempt to zip a 6GB file, and throws the exception then onwards.

Answer 2

Regarding your EDIT: ran into the same issue. After some digging, I discovered that zipFileEntry.Open() returns a WrappedStream , which is the underlying stream (the one that cannot be flushed until finished writing to it).

This WrappedStream is the problem: its max length is of ~2GB. I couldn't find a way to get around this, so I ended up using a different compression library altogether.