Copying files in the same Amazon S3 bucket

Question

I am trying to copy the entire /contentstore/ folder on a bucket to a timestamped version. Basically /contenstore/ would be copied to /contentstore/20130729/ .

My entire script uses s3s3mirror first to clone my production S3 bucket to a backup. I then want to rename the backup to a timestamped copy so that I can keep multiple versions of the same.

I have a working version of this using s3cmd but it seems to take an abnormally long time. The s3s3mirror part between the two buckets is done within minutes, possibly because it is a refresh on existing folder. But even in the case of a clean s3s3mirror (no existing contentstore on backup) it take around 20 minutes.

On the other hand copying the conentstore to a timestamped copy on the backup bucket takes over an hour and 10 minutes.

Am I doing something incorrectly? Should the copy of data on the same bucket take longer than a full clone between two different buckets?

Any ideas would be appreciated.

PS: The command I am running is s3cmd --recursive cp backupBucket/contentStore/ backupBucket/20130729/

Answer 1

Since your source path contains your destination path, you may actually be copying things more than once -- first into the destination path, and then again when that destination path matches your source prefix. This would also explain why copying to a different bucket is faster than within the same bucket.

If you're using s3s3mirror, use the -v option and you'll see exactly what's getting copied. Does it show the same key being copied multiple times?