S3 buckets to Glacier on demand Is it possible from boto3 API?

Question

I'm testing a script to recover date stored in an S3 bucket, with a lifecycle rule that moves the data to glacier each day. So, in theory, when I upload a file to the S3 bucket, after a day, the Amazon infrastructure should move it to glacier.

But I want to test a script that I'm developing in python to test the restore process. So, if I understand the boto3 API I haven't seen any method to force a file stored in the S3 bucket to move immediately to the glacier storage. Is is possible to do that or it's necessary to wait till the Amazon infrastructure fires the lifecycle rule.

I would like to use some code like this:

bucket = s3.Bucket(TARGET_BUCKET)
for obj in bucket.objects.filter(Bucket=TARGET_BUCKET, Prefix=TARGET_KEYS + KEY_SEPARATOR):
    obj.move_to_glacier()

But I can't find any API that make this move to glacier on demand. Also, I don't know if I can force this on demand using a bucket lifecycle rule

Answer 1

Update:

S3 has changed the PUT Object API , effective 2018-11-26. This was not previously possible, but you can now write objects directly to the S3 Glacier storage class.

One of the things we hear from customers about using S3 Glacier is that they prefer to use the most common S3 APIs to operate directly on S3 Glacier objects. Today we're announcing the availability of S3 PUT to Glacier, which enables you to use the standard S3 “PUT” API and select any storage class, including S3 Glacier, to store the data. Data can be stored directly in S3 Glacier, eliminating the need to upload to S3 Standard and immediately transition to S3 Glacier with a zero-day lifecycle policy.

https://aws.amazon.com/blogs/architecture/amazon-s3-amazon-s3-glacier-launch-announcements-for-archival-workloads/

The service now accepts the following values for x-amz-storage-class :

STANDARD
STANDARD_IA
ONEZONE_IA
INTELLIGENT_TIERING
GLACIER
REDUCED_REDUNDANCY

PUT+Copy (which is always used, typically followed by DELETE , for operations that change metadata or rename objects) also supports the new functionality.

Note that to whatever extent your SDK "screens" these values locally, taking advantage of this functionality may require you to upgrade to a more current version of the SDK.

---

This isn't possible. The only way to migrate an S3 object to the GLACIER storage class is through lifecycle policies.

x-amz-storage-class

Constraints: You cannot specify GLACIER as the storage class. To transition objects to the GLACIER storage class, you can use lifecycle configuration.

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

The REST API is the interface used by all of the SDKs, the console, and aws-cli.

---

Note... test with small objects, but don't archive small objects to Glacier in production. S3 will bill you for 90 days minimum of Glacier storage even if you delete the object before 90 days. (This charge is documented.)

Answer 2

It is possible to upload the files from S3 to Glacier using the upload_archive() method of Glacier .

Update: This will not be same as S3 object lifecycle management but direct upload to Glacier.

glacier_client = boto3.client('glacier')

bucket = s3.Bucket(TARGET_BUCKET)

for obj in bucket.objects.filter(Prefix=TARGET_KEYS + KEY_SEPARATOR):
    archive_id = glacier_client.upload_archive(vaultName='TARGET_VAULT',body=obj.get()['Body'].read())
    print obj.key, archive_id

.filter() does not accept the Bucket keyword argument.