How to create one glacier archive for many s3 objects?

Question

One of our client asks to get all the video that they uploaded to the system. The files are stored at s3. Client expect to get one link that will download archive with all the videos.

Is there a way to create such an archive without downloading files archiving it and uploading back to aws?

So far I didn't find the solution.

Is it possible to do it with glacier, or move the files to folder and expose it?

Answer 1

Unfortunately, you can't create a zip-like archives from existing objects directly on S3. Similarly you can't transfer them to Glacier to do this. Glacier is not going to produce a single zip or rar (or any time of) archive from multiple s3 objects for you.

Instead, you have to download them first, zip or rar (or use which ever archiving format you prefer), and the re-upload to S3. Then you can share the zip/rar with your customers.

There is also a possibility of using multi-part AWS API to merge S3 objects without downloading them. But this requires programming custom solution to merge objects (not creating zip/rar-type archives).

Answer 2

You can create a glacier archive for a specific prefix (what you see as a subfolder) by using AWS lifecycle rules to perform this action.

More information available here .

Original answer

There is no native way to be able to retrieve all objects as an archive via S3.

S3 simply exposes all objects as they are uploaded, unfortunately you will need to perform the archiving as a separate process afterwards.