Create a presigned S3 URL for get_object with custom logging information using Boto3?

Question

I am using Boto3 and s3_client.generate_presigned_url to create a presigned get_object url, ie

response = s3_client.generate_presigned_url('get_object',
                                            Params={'Bucket': bucket_name,
                                                    'Key': object_name},
                                            ExpiresIn=expiration)

in accordance with the Boto 3 documentation .

I need to add the identity of the user requesting the URL into the presigned URL itself, so that it will be immediately apparent whose credentials had been used to generate it! Now the AWS Documentation says that it is possible to include custom query string parameters in the URL:

You can include custom information to be stored in the access log record for a request by adding a custom query-string parameter to the URL for the request. Amazon S3 ignores query-string parameters that begin with "x-", but includes those parameters in the access log record for the request, as part of the Request-URI field of the log record. For example, a GET request for s3.amazonaws.com/awsexamplebucket/photos/2019/08/puppy.jpg?x-user=johndoe works the same as the same request for s3.amazonaws.com/awsexamplebucket/photos/2019/08/puppy.jpg , except that the x-user=johndoe string is included in the Request-URI field for the associated log record. This functionality is available in the REST interface only.

The Javascript SDK library has the following recipe in Github issues :

 var req = s3.getObject({Bucket: 'mybucket', Key: 'mykey'}); req.on('build', function() { req.httpRequest.path += '?x-foo=value'; }); console.log(req.presign());

for creating a presigned url with ?x-foo=value embedded into the URL. Judging from the comments it seems to work too!

How do I achieve the same in Python (3), preferably (but not necessarily) using Boto 3?

PS please do note that I am not asking how to force the client to pass a header or anything such - in fact I cannot control the client, I just give the URL to it.

Answer 1

You can use a similar approach in boto3 with botocore events . The events of interest are "provide-client-params.s3.GetObject" and "before-sign.s3.GetObject" . The provide-client-params handler can modify API parameters and context, and before-sign receives among other things the AWSRequest to sign, so we can inject parameters to the URL.

import boto3

from botocore.client import Config
from urllib.parse import urlencode

# Ensure signature V4 mode, required for including the parameters in the signature
s3 = boto3.client("s3", config=Config(signature_version="s3v4"))

def is_custom(k):
    return k.lower().startswith("x-")

def client_param_handler(*, params, context, **_kw):
    # Store custom parameters in context for later event handlers
    context["custom_params"] = {k: v for k, v in params.items() if is_custom(k)}
    # Remove custom parameters from client parameters,
    # because validation would fail on them
    return {k: v for k, v in params.items() if not is_custom(k)}

def request_param_injector(*, request, **_kw):
    if request.context["custom_params"]:
        request.url += "&" if "?" in request.url else "?"
        request.url += urlencode(request.context["custom_params"])

# NOTE: The logic can be included with other client methods as well by simply
# dropping ".GetObject" from the event names
s3.meta.events.register("provide-client-params.s3.GetObject", client_param_handler)
s3.meta.events.register("before-sign.s3.GetObject", request_param_injector)

With the event handlers in place passing custom parameters is a simple matter of including them in the Params dictionary:

response = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': bucket_name,
            'Key': object_name,
            'x-foo': 'value'},
    ExpiresIn=expiration)