Redshift Data Transformation

Question

What is the best way to transform data in Redshift? Like creating a stored procedure that transforms data within the same schema. My background is Oracle using PL/SQL and I used to create functions and procedures as ETL. Now, is there a similar way for Redshift? Tools being used?

Answer 1

While you can have stored procedures in Redshift you will likely be better served by thinking “outside the database”. Redshift is a part of the larger AWS ecosystem and as such operational and orchestration tasks are often better done by different parts of AWS. They will be more powerful and flexible than anything Redshift offers and likely more than what you are use to.

Lambda and Step Functions are my go to for this layer. They are simple and serverless which reduces cost of use and reliability. You may want to look into Redshift Data API as well since this will disconnect the query run time from the Lambda execution time. This also allows for other databases and tools to be part of your updates processes which can be huge.

Classic databases see themselves as the entire operating environment. Redshift is just an engine in the cloud operating environment that is AWS.