使用 boto3 在 S3 中即时提取 7z 文件

Question

I have a really large 7z file in s3 bucket say s3://tempbucket1/Test_For7zip.7z that runs into several tens of GB. I do not want to download it, unzip it and re upload it back to s3. I want to use Boto3 to unzip it on the fly and save it into S3.

I tried to solve this using lzma package based on Previous SO answer which dealt with on the fly unzipping of *.zip files using the fileobj option present in gzip.GzipFile .

from io import BytesIO
import gzip
import lzma
import boto3

# setup constants
bucket = 'tempbucket1'
gzipped_key = 'Test_For7zip.7z'
uncompressed_key = 'Test_Unzip7zip'

# initialize s3 client, this is dependent upon your aws config being done 
s3 = boto3.client('s3', use_ssl=False)  
s3.upload_fileobj(                      # upload a new obj to s3
    Fileobj=lzma.LZMAFile(              
                BytesIO(s3.get_object(Bucket=bucket,
                                      Key=gzipped_key)['Body'].read()),   
                'rb'),                  # read binary
    Bucket=bucket,                      # target bucket, writing to
    Key=uncompressed_key)               # target key, writing to

However, this thows the following error

LZMAError: Input format not supported by decoder

Is there a python package that provides can decode 7z files based on BytesIO, or is there a better way of achieving this?

Answer 1

I never tried this, but Googling gave me this as a possible solution. Please reach out through this post if this solves your problem.