Best way to unpack multi-part archive (zip/rar) in python

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. KyluAce is looking for a . 。

I have 2gb archive (prefer zip or rar) file in parts (let's assume 100parts x 20mb). I didn't find a way to unpack it properly. Firstly I tried with zip. I had files like test.zip, test.z01, test.z02...test.99 when I marge it like that:

    for zipName in zips:
    with open(os.path.join(path_to_zip_file, "test.zip"), "ab") as f:
        with open(os.path.join(path_to_zip_file, zipName), "rb") as z:
            f.write(z.read())

and then after merge unpack it like that:

with zipfile.ZipFile(os.path.join(path_to_zip_file, "test.zip"), "r") as zipObj:
     zipObj.extractall(path_to_zip_file)

I get errors like: test.zip file isn't zip file.

So I tried with rar. Firstly I think that maybe it will be enough when I choose first part and unpack it and it will do it itself but no. So again i merge file (just like in zip case) and then try to unpack it by using patoolib

patoolib.extract_archive("test.rar", outdir="path here")

When I do that I get errors like: patoolib.util.PatoolError: could not find an executable program to extract format rar; candidates are (rar,unrar,7z) patoolib.util.PatoolError: could not find an executable program to extract format rar; candidates are (rar,unrar,7z)

After some work I figured out that this merged files are corrupted (I copied it and try to unpack normally on windows using winrar and there was some problems). So I tried other ways to merge for example using cat cat test.part.* >test.rar but it also doesn't help.

So there is a question there is option to handle it? I need to say that I'm just really disappointed that libs doesn't on their own doesn't unpack multi-part archive it should be naturally.

Answer 1

Calling 7z out of python

1. rename the .zip to .zip.001 and .z01 to zip.002 and so on.
2. call 7z on the 001 ( 7z x test.zip.001 )

import subprocess
cmd = ['7z', 'x', 'test.zip.001']
sp = subprocess.Popen(cmd, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)

CAT

cat test.zip* > test.zip should also work, but not always imho. Tried it for single file and works, but failed with subfolders. Maintaining the right order is mandatory.

Testing:

7z -v1m a test.zip 12MFile
cat test.zip* > test.zip
7z t test.zip
>> Everything is Ok

Can't check with "official" WinRAR (does this even still exist?.) nor WinZIP Files.

Merge File in Python

If you want to stay in python this works too (again for my 7z testfiles..):

import shutil
import glob

with open('output_file.zip','wb') as wfd:
    for f in glob.glob('test.zip.*'): # Search for all files matching searchstring
        with open(f,'rb') as fd:
            shutil.copyfileobj(fd, wfd) # Concatinate

Further remarks

• pyunpack (python frontend) with patool (python backend) and installed unrar or p7zip-rar (7z with the unfree rar-stuff) for linux or 7z in windows can handle zip and rar (and many more) in python
• there is a 7z x -t flag for explicitly set it as split archive (if file is not named 001 maybe helps). Give as eg 7z x -trar.split or 7z x -tzip.split or something.