有效压缩具有许多相同文件的文件系统目录树

Question

We have multiple .NET web applications all sharing quite a few common libraries. None of them are in the GAC.

The deployment constraint is that all of these web applications have dedicated directories. Which results in large amount of duplicated dlls in the total directory structure.

This directory structure is extracted from a single zip archive.

As a result the zip archive has many identical files found in different directories.

This is huge redundancy, which I want to eliminate in the zip archive, I do not care much if redundant files are created on the disk. I see two ways optimize the zip:

1. Use windows symbolic links and junctions to reduce the amount of physical identical files.
2. Use smart compression that would not compress the same file data twice.

Method 1

I used zip and 7z to test compressing directory structures. I used junctions and file symbolic links as the means to reduce space on disk.

Unfortunately, both zip and 7z compress junctions as if they were full blown directories. A symbolic link is compressed as a zero length file by 7z, its nature as a symbolic link is lost upon decompression. zip traverses the symbolic link and compresses the target data instead, which results in duplicate file content in the archive.

In short I failed to eliminate the duplicate file data using the first method.

Method 2

What I want is exactly described by http://sourceforge.net/p/sevenzip/feature-requests/794/ . However, it is nothing more than a feature request.

A comment to the feature request mentions lrzip as an efficient huge file compressor. I have to check it, but it does not seem to eliminate duplicate file data the way I would like it to be.

Any help is welcome.

Answer 1

mark, how did you try lrzip? It can't detect duplicates inside compressed archive (default zip); it should be used with some non-compressing archive (in Unix world - with tar) or zipfile created without compression (you will get archive with the size almost equal to sum of input sizes).

You can also try any multi-file compressor, capable of solid mode (rar, 7z), but this may not work if your archive is huge and there is big distance between duplicates. lrzip supports greater distance.

Tar (and PAX) on Unix supports hard and soft links: http://www.gnu.org/software/tar/manual/html_section/tar_71.html#SEC140