压缩100个存储在S3上的文件

Question

We use S3 to store various media uploaded via our application such as images, documents etc. We work in the property software industry and as a means of exchanging data stored in our system with property portals a common exchange format between portals is the Rightmove BLM data feed specification. This is essentially a zip file containing a delimited text file and any associated media which is sent via FTP to each portal. However a bottleneck in the process is downloading the media from S3 for zipping. For example one single account on our system could have in the region of 1000 images/documents to the downloaded and zipped in preparation for transfer (each file has to be name in a particular format for that particular portal (unique number, sequence numbers etc). However downloading 1000 images/documents from S3 to an EC2 server in the same region via the PHP SDK takes some time (60+ seconds). When doing this for multiple accounts at the same time it puts considerable load on the server.

Is there an better/faster way to download files from S3 so they can be prepped and zipped on the EC2 instance?

Thanks.

Answer 1

One option would be to aggregate the zip as the files are added. Meaning, instead of zipping the files all at once, use a Lambda function to add them to a zip file as they're added to or updated on the S3 bucket. Then, the zip would be available more-or-less on demand.