[英]How to read from a text file compressed with 7z?
I would like to read (in Python 2.7), line by line, from a csv (text) file, which is 7z compressed.我想从 7z 压缩的 csv(文本)文件中逐行读取(在 Python 2.7 中)。 I don't want to decompress the entire (large) file, but to stream the lines.我不想解压缩整个(大)文件,而是要流式传输这些行。
I tried pylzma.decompressobj()
unsuccessfully.我尝试pylzma.decompressobj()
失败。 I get a data error.我收到数据错误。 Note that this code doesn't yet read line by line:请注意,此代码尚未逐行读取:
input_filename = r"testing.csv.7z"
with open(input_filename, 'rb') as infile:
obj = pylzma.decompressobj()
o = open('decompressed.raw', 'wb')
obj = pylzma.decompressobj()
while True:
tmp = infile.read(1)
if not tmp: break
o.write(obj.decompress(tmp))
o.close()
Output:输出:
o.write(obj.decompress(tmp))
ValueError: data error during decompression
This will allow you to iterate the lines.这将允许您迭代这些行。 It's partially derived from some code I found in an answer to another question.它部分源自我在另一个问题的答案中找到的一些代码。
At this point in time ( pylzma-0.5.0
) the py7zlib
module doesn't implement an API that would allow archive members to be read as a stream of bytes or characters — its ArchiveFile
class only provides a read()
function that decompresses and returns the uncompressed data in a member all at once.此时 ( pylzma-0.5.0
) py7zlib
模块没有实现允许将存档成员作为字节或字符流读取的py7zlib
它的ArchiveFile
类只提供了一个read()
函数,用于解压缩和一次性返回成员中未压缩的数据。 Given that, about the best that can be done is return bytes or lines iteratively via a Python generator using that as a buffer.鉴于此,可以做的最好的事情是通过 Python 生成器将其用作缓冲区,以迭代方式返回字节或行。
The following does the latter, but may not help if the problem is the archive member file itself is huge.以下是后者,但如果问题是存档成员文件本身很大,则可能无济于事。
The code below should work in Python 3.x as well as 2.7.下面的代码应该适用于 Python 3.x 和 2.7。
import io
import os
import py7zlib
class SevenZFileError(py7zlib.ArchiveError):
pass
class SevenZFile(object):
@classmethod
def is_7zfile(cls, filepath):
""" Determine if filepath points to a valid 7z archive. """
is7z = False
fp = None
try:
fp = open(filepath, 'rb')
archive = py7zlib.Archive7z(fp)
_ = len(archive.getnames())
is7z = True
finally:
if fp: fp.close()
return is7z
def __init__(self, filepath):
fp = open(filepath, 'rb')
self.filepath = filepath
self.archive = py7zlib.Archive7z(fp)
def __contains__(self, name):
return name in self.archive.getnames()
def readlines(self, name, newline=''):
r""" Iterator of lines from named archive member.
`newline` controls how line endings are handled.
It can be None, '', '\n', '\r', and '\r\n' and works the same way as it does
in StringIO. Note however that the default value is different and is to enable
universal newlines mode, but line endings are returned untranslated.
"""
archivefile = self.archive.getmember(name)
if not archivefile:
raise SevenZFileError('archive member %r not found in %r' %
(name, self.filepath))
# Decompress entire member and return its contents iteratively.
data = archivefile.read().decode()
for line in io.StringIO(data, newline=newline):
yield line
if __name__ == '__main__':
import csv
if SevenZFile.is_7zfile('testing.csv.7z'):
sevenZfile = SevenZFile('testing.csv.7z')
if 'testing.csv' not in sevenZfile:
print('testing.csv is not a member of testing.csv.7z')
else:
reader = csv.reader(sevenZfile.readlines('testing.csv'))
for row in reader:
print(', '.join(row))
If you can use python 3, there is a useful library, py7zr , which supports partially 7zip decompression as below:如果您可以使用 python 3,则有一个有用的库py7zr ,它支持部分7zip 解压,如下所示:
import py7zr
import re
filter_pattern = re.compile(r'<your/target/file_and_directories/regex/expression>')
with SevenZipFile('archive.7z', 'r') as archive:
allfiles = archive.getnames()
selective_files = [f if filter_pattern.match(f) for f in allfiles]
archive.extract(targets=selective_files)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.