Transforming data in AWS Kinesis Firehose with AWS Lambda / CloudFormation

Question

I found this guide on the AWS blog that illustrates an example of what I am trying to accomplish. I currently have a workflow that looks like Kinesis Stream --> Kinesis Firehose --> S3 bucket, and I want to introduce a Lambda where I can transform the data before it reaches the final destination.

First, the links for the Lambda blueprints don't work on that article. Nor the official documentation for the data transformation in Firehose . Does anyone have a working Python blueprint for this case?

Secondly, the guide shows that when creating a Firehose Delivery Stream in the AWS Console, there is an option to "Enable" data transformation in Lambda. However, I do not see any property for AWS::KinesisFirehose::DeliveryStream in the CloudFormation documentation that would accomplish the same thing. How would I enable this in my CloudFormation template?

Thanks!

Answer 1

For the CF issue, you're after the ExtendedS3DestinationConfiguration and more specifically the ProcessingConfiguration

Here's a simple python processor