Boto3 S3 update metadata of existing object

Question

This is a strange thing that I can't wrap my head around just yet. Why is it that when I use Boto3 to put an "expires" datetime on an object that gets put to AWS S3 by put_object, it gets stored and shows in the AWS cosnole as "metadata." However, when I retrieve the object my "expires" datetime shows up as a datetime element of the object rather than a datetime element in the Metadata dictionary.

This question puzzled me but I worked around it without understanding it. Now it comes to me that using this method: How to update metadata of an existing object in AWS S3 using python boto3? which is copied below for ease of reading:

import boto3

s3 = boto3.resource('s3')
s3_object = s3.Object('bucket-name', 'key')
s3_object.metadata.update({'id':'value'})
s3_object.copy_from(
    CopySource={'Bucket':'bucket-name', 'Key':'key'}, 
    Metadata=s3_object.metadata, MetadataDirective='REPLACE')

Causes my "expires" metadata to be destroyed. Of course I tried this:

metakeys.metadata.update({'x-amz-meta-hell':'yes', 'expires': metakeys.expires})

But that throws: AttributeError: 'datetime.datetime' object has no attribute 'encode'

It is true that you can update the metadata effectively without destroying the "expires" element through the console. So to some extent I am suggesting that the method above is either A: Not viable or not correct, B: Broken, or C: both broken and not correct

The question is - what is the correct way to update metadata of an object without destroying this or future odd behaviors of AWS S3 objects?

Answer 1

If you do a put_object() with "Expires" parameter, you should get something like this.

{
    'Expiration': 'string',
    'ETag': 'string',
    ..........
}

However, the Expiration is an attributes, it is NOT your user custom metadata. All user custom meta data can only be STRING, and all will carry a prefix x-amz-meta- when your check the metadata.

metakeys.metadata.update({'x-amz-meta-hell':'yes', 'expires': metakeys.expires})

Above update will failed, if the given metakeys.expires is not string. it can be as simple as using isoformat() to convert it to string.

Although copy_object() allow you specify explicit expiration datetime, HOWEVER, the API documentation doesn't explicit mentioned that original file expiration datetime will be copy over to target object.