Will AWS S3 event forwarding to SQS will end up being a different message than S3 to SNS to SQS?

Question

I am using a Splunk Technical Add-on that will be pulling messages from an SQS queue. Although the TA suggests using S3 forwarding to an SNS and it subscribed to an SQS, there is also the possibility of S3 to forward directly to SQS.

Would SNS make any change on what S3 send to it? Or would it be a fully transparent transport method to SQS?

Answer 1

Yes, by default, S3 → SQS and S3 → SNS → SQS will result in two different data structures/formats inside the SQS message body.

This is because an SNS subscription provides metadata with each message delivered -- the SNS MessageId , a Signature to validate authenticity, a Timestamp of when SNS originally accepted the message, and other attributes. The original message is encoded as a JSON string inside the Message attribute of this outer JSON structure.

So with SQS direct, you would extract the S3 event with (pseudocode)...

s3event = JSON.parse(sqsbody)

...but with SNS to SQS...

s3event = JSON.parse(JSON.parse(sqsbody).Message)

You can disable the additional structures and have SNS send only the original payload by enabling raw message delivery on the SQS subscription to the SNS topic.

https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html

With raw message delivery enabled, the contents become the same, for both S3 → SQS and S3 → SNS → SQS.

The downside to raw message delivery is that you lose potentially useful troubleshooting information with raw message delivery, like the SNS message ID and SNS-issued timestamp.

On the other hand, if the receiving service (the SQS consumer) assumes the messages are always coming via SNS and expects to find the SNS data structure in the SQS message body, then sending direct S3 → SQS will result in the consumer finding that the message body from SQS does not match its expectations.