How do I download all the versions of a file with 100,000+ versions from Amazon S3?

Question

I'm using the AWS command line on Windows and all methods I have found till now seem to suggest I need to get a list of version-ids for all the objects. Is there some sort of wildcard like * that I can use?

Answer 1

This Python code using boto will download all versions of files found in a bucket. It's possible that a large number of versions will require paging through the result set.

import boto
conn = boto.connect_s3()
bucket = conn.get_bucket('BUCKET')

# Get a list of all versions contained in the bucket
versions = bucket.list_versions(prefix='FILENAME')

for v in versions:
  # Save the version to a filename based on the Last Modified date
  v.get_contents_to_filename(v.last_modified)

Answer 2

With Boto3 the solution from John needs to be updated as below. I am saving the files with the modified ts.

import boto3
client = boto3.client('s3')

_bucket = '<s3Bucket>'
_file   = '<fileName>'
_key    = '<the s3 prefix>' + _file
_local  = '<local path>' + _file

response = client.list_object_versions(
    Bucket=_bucket,
    Prefix=_key
)

for v in response['Versions']:
    client.download_file(_bucket, _key,
                         _local + '_' + v['LastModified'].strftime('%Y%m%d%H%M%S'),
                         ExtraArgs={"VersionId": v["VersionId"]})
    print(v['LastModified'])