简体   繁体   中英

Stream/Task can be used as ETL in warehouse solution or not

I want to replicate the the ETL solution is snowflake. I have achieved that using streams and tasks. I am using stored procedure on the stream table where there are merge queries that replicate the logic in any other ETL tool like Informatica and tasks to automate the flow. Is this approach feasible for delta changes?

IMO, an ETL architecture consisting Snowflake tools (Snowpipe, Streams, Tasks, Stored Procedures) is a perfectly fine solution for many/most data pipeline use cases. I've now seen both small and large organizations with multiple in-house ETL tools use this architecture within Snowflake, quite frankly because it works and it's all they need for their use case.

I'm sure there are some use cases where moving forward with your favorite ETL tool make sense, but if you're comfortable with this architecture and it meets your need, there is no reason to not use it.

A good diagram of the data pipeline architecture can be found at this link:

https://docs.snowflake.com/en/user-guide/data-pipelines-intro.html

If you need more information on how to use streams, this link is the place to start:

https://docs.snowflake.com/en/user-guide/streams.html

If you need more information on your MERGE command, the docs are here:

https://docs.snowflake.com/en/sql-reference/sql/merge.html

And there is a nice blog post on MERGE here:

https://dwgeek.com/snowflake-merge-statement-syntax-usage-and-examples.html/

I hope this helps...Rich

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM