Can you add SQS message attributes to SQS messages generated by S3 events?

Question

I'd like to use AWS S3 to allow some users to add files to an S3 bucket.

Then, I'd like to generate an SQS message when a new file has been added.

Last, I'd like to consume the SQS message and process it with a background job worker of a particular class.

I'd like to use SQS message attributes to determine which background job worker class I should use for processing. As the SQS message attribute documentation states:

Message attributes [...] can be used by the consumer of the message to help decide how to handle the message without having to first process the message body.

(via the SQS Developer Guide )

However, under the S3 bucket's Properties, under Advanced Settings, the Events settings do not appear to expose a way to set message attributes.

Is there a way to specify message attributes on an event-by-event basis for events generated from S3?

Answer 1

You should use a Lambda to add messages to the queue, for example with python using BOTO 3 API. Map this Lambda to the S3 Event. http://docs.aws.amazon.com/lambda/latest/dg/with-s3.html

Answer 2

There is not a way to inject custom message attributes into S3 event notifications... but also note that you may be misinterpreting what message attributes can be used for.

Message attributes [...] can be used by the consumer of the message

This means they provide a mechanism for the consumer to triage a message after the consumer has already received it from the queue.

You can't selectively consume messages based on message attributes. Queue consumers receive the next available message(s) when they poll the queue. They don't get to select which messages they consume.

If you want to divide the messages up by class, you'll need an intermediate process that selectively distributes messages to the appropriate (separate) downstream queues. Better, if your structure allows it, might be different event configurations matching specific patterns that need to go to individual queues.