AWS Synchronize S3 bucket with EC2 instance

Question

I would like to synchronize an S3 bucket with a single directory on multiple Windows EC2 instances. When a file is uploaded or deleted from the bucket, I would like it to be immediately pushed or removed respectively from all of the instances. New instances will be added/removed frequently (multiple times per week). Files will be uploaded/deleted frequently as well. The files sizes could be up to 2gb in size. What AWS services or features can solve this?

Answer 1

Based on what you've described, I'd propose the following solution to this problem.

You need to create an SNS topic for S3 change notifications. Then you need a script that's going to subscribe to this topic from your machines. This script will update files on your machines based on changes coming from S3 . It should support basic CRUD operations.

Run this script and then sync contents of your S3 to your machine when it starts using aws-cli mentioned above.

Answer 2

Yes, i have used the aws cli s3 "sync" command to keep a local server's content updated with S3 changes. It allows a local target directory's files to be synchronized with a bucket or prefix. http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

Answer 3

Edit : This following answer is to sync EC2 with S3 Bucket, Source : EC2 & Destination : Bucket.

If it were for only one instance, then only aws cli sync (with --delete option) would have been worked for both: putting files to S3 bucket and to delete.

But the case here is for Multiple Instances , so if we use aws s3 sync with --delete option, there would be a problem.

To explain it simply, consider Instance I1 with files a.jpg & b.jpg to be synced to Bucket. Now a CRON job has synced the files with the S3 bucket.

Now we have Instance I2 which has files c.jpg & d.jpg . So when the CRON job of this Instance runs, it puts the files c.jpg & d.jpg and also deletes the files a.jpg & b.jpg , because those files doesn't exist in Instance I2 .

So to rectify the problem we have two approaches :

1. Sync all files across all Instances(Costly and removes the purpose of S3 altogether).
2. Sync files without the --delete option, and implement the deletion separately(using aws s3 rm ).