下载时解压缩存档

Question

I've got a program that downloads part01, then part02 etc of a rar file split across the internet. My program downloads part01 first, then part02 and so on. After some tests, I found out that using, on example, UnRAR2 for python I can extract the first part of the file (an.avi file) contained in the archive and I'm able to play it for the first minutes. When I add another file it extracts a bit more and so on. What I wonder is: is it possible to make it extract single files WHILE downloading them? I'd need it to start extracting part01 without having to wait for it to finish downloading... is that possible?

Thank you very much!

Matteo

Answer 1

You are talking about an.avi file inside the rar archives. Are you sure the archives are actually compressed? Video files released by the warez scene do not use compression:

Ripped movies are still packaged due to the large filesize, but compression is disallowed and the RAR format is used only as a container. Because of this, modern playback software can easily play a release directly from the packaged files, and even stream it as the release is downloaded (if the network is fast enough).

(I'm thinking VLC, BSPlayer, KMPlayer, Dziobas Rar Player, rarfilesource, rarfs,...) You can check for the compression as follows:

• Open the first.rar archive in WinRAR. (name.part01.rar or name.rar for old style volumes names )
• Click the info button.

If Version to extract indicates 2.0, then the archive uses no compression. (unless you have decade old rars) You can see Total size and Packed size will be equal.

is it possible to make it extract single files WHILE downloading them?

Yes. When no compression is used, you can write your own program to extract the files. (I know of someone who wrote a script to directly download the movie from external rar files; but it's not public and I don't have it.) Because you mentioned Python I suggest you take a look at rarfile 2.2 by Marko Kreen like the author of pyarrfs did. The archive is just the file chopped up with headers (rar blocks) added. It will be a copy operation that you need to pause until the next archive is downloaded.

I strongly believe it is also possible for compressed files. Your approach here will be different because you must use unrar to extract the compressed files. I have to add that there is also a free RARv3 implementation to extract rars implemented in The Unarchiver.

I think this parameter for (un)rar will make it possible:

 -vp Pause before each volume By default RAR asks for confirmation before creating or unpacking next volume only for removable disks. This switch forces RAR to ask such confirmation always. It can be useful if disk space is limited and you wish to copy each volume to another media immediately after creation.

It will give you the possibility to pause the extraction until the next archive is downloaded.

I believe that this won't work if the rar was created with the 'solid' option enabled.

When the solid option is used for rars, all packed files are treated as one big file stream . This should not cause any problems if you always start from the first file even if it doesn't contain the file you want to extract. I also think it will work with passworded archives.

Answer 2

I highly doubt it. By nature of compression (from my understanding), every bit is needed to uncompress it. It seems that the source of where you are downloading from has intentionally broken the avi into pieces before compression, but by the time you apply compression, whatever you compressed is now one atomic unit. So they kindly broke the whole avi into Parts, but each Part is still an atomic nit.

But I'm not an expert in compression.

The only test I can currently think of is something like: curl http://example.com/Part01 | unrar curl http://example.com/Part01 | unrar .

Answer 3

I don't know if this was asked with a specific language in mind, but it is possible to stream a compressed RAR directly from the internet and have it decompressed on the fly. I can do this with my C# library http://sharpcompress.codeplex.com/

The RAR format is actually kind of nice. It has headers preceding each entry and the compressed data itself does not require random access on the stream of bytes.

Do it multi-part files, you'd have to fully extract part 1 first, then continue writing when part 2 is available.

All of this is possible with my RarReader API. Solid archive are also streamable (in fact, they're only streamable. You can't randomly access files in a solid archive. You pretty much have to extract them all at once.)