AWS Redshift ETL Process

Question

I'm investigating redshift for our Data Warehouse, and I'm trying to think of how to architect a solution.

I have an instance of Amazon Kinesis Firehose as a delivery stream which writes to my Redshift database, and all that works fine.

Now my issue is how do I automate the creation of dimensions and fact tables.

Can I use a Lambda function in the delivery stream to write to the fact table and update the dimensions?

Answer 1

The Data Transformation capability of AWS Lambda on an Amazon Kinesis Firehose is purely to modify or exclude streaming data. It cannot be used to create other tables.

If you wish to create dimension and fact tables, or otherwise perform ETL, you'll need to trigger it externally, such as having a scheduled task run SQL commands on your Amazon Redshift instance. This task would connect via JDBC/ODBC to run the commands.