Python - 并行解压缩.gz文件

Question

I have multiple .gz files that add up to 1TB in total. How can I utilize Python 2.7 to unzip these files in parallel? looping on the files takes too much time.

I tried this code as well:

filenames = [gz for gz in glob.glob(filesFolder + '*.gz')]

def uncompress(path):
    with gzip.open(path, 'rb') as src, open(path.rstrip('.gz'), 'wb') as dest:
        shutil.copyfileobj(src, dest)

with multiprocessing.Pool() as pool:
    for _ in pool.imap_unordered(uncompress, filenames, chunksize=1):
        pass

However I get the following error:

  with multiprocessing.Pool() as pool:

AttributeError: __exit__

Thanks!

Answer 1

To use with construct, the object used inside must have __enter__ and __exit__ methods. The error says that the Pool class (or instance) doesn't have these so you can't use it in the with statement. Try this (just removed the with statement):

import glob, multiprocessing, shutil

filenames = [gz for gz in glob.glob('.' + '*.gz')]

def uncompress(path):
    with gzip.open(path, 'rb') as src, open(path.rstrip('.gz'), 'wb') as dest:
        shutil.copyfileobj(src, dest)


for _ in multiprocessing.Pool().imap_unordered(uncompress, filenames, chunksize=1):
    pass

EDIT

I agree with @dhke, unless all (or most) of gz files are physically located adjacently, frequent disk reads for different locations (which are called more frequently when using multiprocessing) will be slower as compared to doing these operations file by file one by one (serially).