Workflow for updating assets on Amazon S3 and CloudFront

Question

I am trying to build a workflow to update files on a S3 bucket and invalidate them on Cloudfront so it gets removed from its cache.

These files consist of JS, CSS, images, media, etc. I am using grunt to minify them.

This is what an ideal scenario in my opinion would be:

1. run grunt on the latest codebase to prepare for distribution;
2. upload new files from step 1 to S3 using aws client tools;
3. invalidate these new files on Cloudfront using aws client tools.

The problem I'm facing is that, on step 1, the minified files all have a newer timestamp than what's on S3, so when I run aws s3 sync, it will try to upload all the files back to S3. I just want the modified files to be uploaded.

I'm open to suggestions on changing the entire workflow as well. Any suggestions?

Answer 1

s3cmd would be able to solve the problem with uploading only those files which have been modified. Rather than checking for timestamp changes , it checks for content changes (internally it assigns MD5 hashes to each file and then checks the local version of the file with the one present at S3, uploading only those files whose MD5 hashes mismatch)

It has many command line options including options to invalidate uploaded files from CloudFront distribution