尝试使用Python和smart_open动态解压缩大文件时出现“EOFError：已到达流末尾”错误

Question

I'm trying to download and decompress a set of files from a remote Apache server. I provide a list of .tbz (tar.bz2) files to be downloaded and decompressed on the fly. The goal is to stream them from the remote Apache server through the tar decompressor and immediately stream them to my Amazon AWS S3 bucket. I do this because files can be as large as 30Gb.

I use the "smart_open" python library to abstract away https and s3 management.

The code I provide here works fine for small files. As soon as I'm trying to do this with a larger file (over 8Mb), I get the following error:

"EOFError: End of stream already reached"

Here's the traceback:

Traceback (most recent call last):
  File "./script.py", line 28, in <module>
    download_file(fileName)
  File "./script.py", line 21, in download_file
    for line in tfext:
  File "/.../lib/python3.7/tarfile.py", line 706, in readinto
    buf = self.read(len(b))
  File "/.../lib/python3.7/tarfile.py", line 695, in read
    b = self.fileobj.read(length)
  File "/.../lib/python3.7/tarfile.py", line 537, in read
    buf = self._read(size)
  File "/.../lib/python3.7/tarfile.py", line 554, in _read
    buf = self.cmp.decompress(buf)
EOFError: End of stream already reached

When I print out the lines I'm writing to the stream, I can see that I'm still getting through the first fraction of the file before the error is being thrown.

What I've tried so far:

1. I've tried to specify the same buffer size for both open() and tarfile.open() without success.

2. I've also tried to introduce some delay between writing of each line to no avail either.

from smart_open import open
import tarfile

baseUrl = 'https://someurlpath/'
filesToDownload = ['name_of_file_to_download']

def download_file(fileName):
    fileUrl = baseUrl + fileName + '.tbz'
    with open(fileUrl, 'rb') as fin:
        with tarfile.open(fileobj=fin, mode='r|bz2') as tf:
            destination = 's3://some_aws_path/' + fileName + '.csv'
            with open(destination, 'wb') as fout:
                with tf.extractfile(tf.next()) as tfext:
                    for line in tfext:
                        fout.write(line)


for fileName in filesToDownload:
    download_file(fileName)

I want to be able to process large files exactly the same way I'm able to process small ones.

Answer 1

The compressed tar extraction requires file seek which may not be possible with virtual file descriptor as created by smart_open. Alternative is to download data to block storage before processing.

from smart_open import open
import tarfile
import boto3
from codecs import open as copen

filenames = ['test.tar.bz2',]

def download_file(fileName):
   s3 = boto3.resource('s3')
   bucket = s3.Bucket('bucketname')
   obj = bucket.Object(fileName)
   local_filename = '/tmp/{}'.format(fileName)
   obj.download_file(local_filename)
   tf = tarfile.open(local_filename, 'r:bz2')
   for member in tf.getmembers():
      tf.extract(member)
      fd = open(member.name, 'rb')
      print(member, len(fd.read()))
if __name__ == '__main__':
   for f in filenames:
      download_file(f)