Compressing files with PowerShell throws OutOfMemory Exception

Question

I am using PowerShell v5 and trying to archive the file with the Compress-Archive cmdlet from Microsoft.PowerShell.Archive module:

Compress-Archive -LiteralPath $GLBSourcePathFull -CompressionLevel Optimal -DestinationPath $GLBArchiveFile

This worked flawlessly with 3 files, which had the following sizes: 16MB , 341MB and 345MB .

However once it came across the files bigger in size than 600MB (approximately), PowerShell threw the following exception:

Exception calling "Write" with "3" argument(s): "Exception of type 'System.OutOfMemoryException' was thrown."

The same thing happened with files over 1GB in size.

To add more context to my situation, I am trying to zip up the file from the local folder to one of the network locations within my company, however I doubt there is a difference as I tested this on my local PC just to get the same results.

Have you ever encountered this before? Is it trying to read the whole file into memory before outputting the zip instead of writing directly to the disk? Or maybe there is a limit to how much memory PowerShell can use by default?

I know there are a few other solutions like 7Zip4powerShell module, but I am not allowed to use anything open source at this point, so I would like to understand the current situation I have and how I could potentially address this.

Thank you for any comments you may have.

Answer 1

Compress-Archive cmdlet probably uses a naive approach of loading/mapping the entire source file and target archive into memory and since you're apparently using 32-bit PowerShell these files (along with the PowerShell process code and other data used by the process) don't fit into the process address space which is 2GB (or 3-4GB if it's LARGEADDRESSAWARE).

64-bit PowerShell on a machine with lots of RAM (eg 32GB) successfully compresses 1GB+ files.

If you're stuck with 32-bit PowerShell and the built-in cmdlet, try splitting the file into 100MB files and use some descriptive file names to be able to join them in your unpacking script. Obviously such an archive would be unusable for anyone without the re-assembling script.