在python中通过块解压缩文件夹

Question

I have a big zip file containing many files that i'd like to unzip by chunks to avoid consuming too much memory.

I tried to use python module zipfile but I didn't find a way to load the archive by chunk and to extract it on disk.

Is there simple way to do that in python ?

EDIT

@steven-rumbalski correctly pointed that zipfile correctly handle big files by unzipping the files one by one without loading the full archive.

My problem here is that my zip file is on AWS S3 and that my EC2 instance cannot load such a big file in RAM so I download it by chunks and I would like to unzip it by chunk.

Answer 1

You don't need a special way to extract a large archive to disk. The source Lib/zipfile.py shows that zipfile is already memory efficient. Creating a zipfile.ZipFile object does not read the whole file into memory. Rather it just reads in the table of contents for the ZIP file. ZipFile.extractall() extracts files one at a time using shutil.copyfileobj() copying from a subclass of io.BufferedIOBase .

If all you want to do is a one-time extraction Python provides a shortcut from the command line:

python -m zipfile -e archive.zip target-dir/

Answer 2

You can use zipfile (or possibly tarfile) as follows:

import zipfile

def extract_chunk(fn, directory, ix_begin, ix_end):
    with zipfile.ZipFile("{}/file.zip".format(directory), 'r') as zf:
        infos = zf.infolist()
        print(infos)
        for ix in range(max(0, ix_begin), min(ix_end, len(infos))):
                zf.extract(infos[ix], directory)
        zf.close()

directory = "path"
extract_chunk("{}/file.zip".format(directory), directory, 0, 50)