在 AWS S3 存储桶上组合 7z 块

Question

I have compressed a file into several chunks using 7zip:

HAVE - the following saved in an AWS S3 Bucket:

pic.7z.001
pic.7z.002
pic.7z.003
pic.7z.004
pic.7z.005

Each chunk is 700 MB.

WANT - the following also saved in an AWS S3 Bucket:

first step: pic.7z
second step: a folder called 'pic'

I want to do it all on the AWS S3 if possible.

How do I unzip and combine these chunks to get a single file using python?

Answer 1

It's not possible to do this in S3, since S3 is only responsible for storage. You can do this in AWS lambda. You can invoke one lambda function per chunk and load the chunk into memory since the max disk space is only 512 MB for each function. Then initialize a multipart upload, and use the S3 multipart upload functionality to upload each chunk to S3. Finally, complete the multipart upload and S3 will automatically concatenate all the parts together into one 7z file.