在python中逐行读取一个大的压缩文本文件

Question

I am trying to use zipfile module to read a file in an archive. the uncompressed file is ~3GB and the compressed file is 200MB. I don't want them in memory as I process the compressed file line by line. So far I have noticed a memory overuse using the following code:

import zipfile
f = open(...)
z = zipfile.ZipFile(f)
for line in zipfile.open(...).readlines()
  print line

I did it in C# using the SharpZipLib:

var fStream = File.OpenRead("...");
var unzipper = new ICSharpCode.SharpZipLib.Zip.ZipFile(fStream);
var dataStream =  unzipper.GetInputStream(0);

dataStream is uncompressed. I can't seem to find a way to do it in Python. Help will be appreciated.

Answer 1

Python file objects provide iterators, which will read line by line. file.readlines() reads them all and returns a list - which means it needs to read everything into memory. The better approach (which should always be preferred over readlines() ) is to just loop over the object itself, Eg:

import zipfile
with zipfile.ZipFile(...) as z:
    with z.open(...) as f:
        for line in f:
            print line

Note my use of the with statement - file objects are context managers, and the with statement lets us easily write readable code that ensures files are closed when the block is exited (even upon exceptions). This, again, should always be used when dealing with files.

Answer 2

If the inner directory and the subdirectory filenames in the zipped file don't matter, you can try this:

from zipfile import ZipFile
from io import TextIOWrapper

def zip_open(filename):
    """Wrapper function that for zipfiles."""
    with ZipFile(filename) as zipfin:
        for filename in zipfin.namelist():
            return TextIOWrapper(zipfin.open(filename))

# Usage of the zip_open function)
with zip_open('myzipball.zip') as fin:
    for line in fin:
        print(line)

The zip_open works well when the zipfile contains a single or multiple files without subdirectories. Not sure if the simple for filename in zipfin.namelist() works if there are complex subdirectories structure in the zipped file though.