如何以经济高效的方式将日常构建存储在Amazon S3中？

Question

I'm trying to make a daily build machine using EC2 and store the daily releases in S3.

The releases are complete disk images so they are very bloated(300+MB total, 95% OS kernel/RFS/libraries, 5% actual software). And they change very little across time.

Ideally, with good compression, the storage cost should be close to O(t) , t for time.

But if I simply add those files to S3 every day, with version number as part of file name, or with the same file name each time but with the S3 bucket versioned, the cost would be O(t^2) .

Because according to this , all versions takes space and I'm charged for the space a new version takes ever since a new version is created.

Glacier is cheaper but still O(t^2) .

Any suggestions?

Answer 1

Basically what you're looking for is an incremental file-level backup. (ie only backup things that change) and rebuild the current state by using a full backup and applying the deltas (ie increments).

If you need to use the latest image you probably need to do incremental + keep latest image. You also probably want to do full backups from time to time to reduce the time it takes to rebuild from incremental (and you are going to need to keep some sort of metadata associated with the backups).

So to sum it up: what you are describing is possible, you just need to do extra work apart from just pushing the image. Presumably you have a build process that generates the image an the extra steps can be inserted between generation and upload. The restore process is going to be more complicated than currently.

To get you started look at binary diff tools like bsdiff/bspatch or xdelta. You could generate the delta and back up only the delta. The image is also compressed so if you diff the compressed versions you will not get very far, so you probably want to diff the uncompressed file. Another way to look at it is to do the diff before generating an image and picking up only files that changed (probably more complex)