C# OutOfMemoryException creating ZipOutputStream using SharpZipLib

Question

I keep getting a very annoying OutOfMemory exception on the following code.

I'm zipping a lot of small files (PDF, each being 1.5mb approx).

At first I was getting the exception afer 25 files zipped, which doesn't seem like a massing archive.

Setting up the size of the ZipEntry somehow helped since now I manage to get up to 110 files zipped (I'm debugging under visual studio)

Here's my code, maybe there's something wrong with it.

Any help would be greatly appreciated.

Thanks

    public static MemoryStream Zip(Dictionary<string, byte[]> files)
    {
        var outputMemStream = new MemoryStream();

        var zipStream = new ZipOutputStream(outputMemStream);

        zipStream.SetLevel(9);
        foreach (var file in files)
        {
            zipStream.PutNextEntry(new ZipEntry(file.Key.FmtValidFileName())
                {
                    Size = file.Value.Length
                });
            zipStream.Write(file.Value, 0, file.Value.Length);
            zipStream.Flush();
        }           
        zipStream.Finish();
        outputMemStream.Position = 0;
        return outputMemStream;
    }

Answer 1

As always, a concise-but-complete code example would go a long way toward helping you get good answers.

That said, you might want to consider using the (relatively new) System.IO.Compression.ZipArchive class in .NET. It is potentially less buggy and/or more reliable than third party libraries (though I admit SharpZipLib is reasonably well-respected :) ).

More important, you can instantiate a new ZipArchive object with the ZipArchiveMode.Create value, which will cause the compressed data to be written directly to the stream rather than being cached in-memory. In this mode, out-of-memory errors should be non-existent, no matter how much data or how many archive items you're trying to create.

EDIT: One more thing: to ensure completely against out-of-memory problems, make sure that whatever .zip implementation you are using, you write directly to the disk. Writing to a temporary in-memory MemoryStream will of course impose limitations on your process that otherwise need not occur.

Answer 2

I gave up trying to use MemoryStream even though being on a 64bit system with 16gb of memory I should have been safe on that side.

The relevant topic I found was: OutOfMemoryException while populating MemoryStream: 256MB allocation on 16GB system

And using a temporary file to write/read the data instead of memory.