云数据融合管道中的 SCD 类型 2 实现

Question

I have requirement to apply SCD2 in my data fusion pipeline and my source data is GCS and target is BigQuery. I have created the pipeline and able to load the data into target BigQery so now I want to implement merge and update process as part of SCD2. Can you please help me as I'm new to Google cloud data fusion.

Answer 1

@phaneendra's approach is a possible solution. Anyhow, Data Fusion is designed to be used in a code-free visual way. If you want to take advantage of its potential, I would recommend you to use Wrangler for your data transformations. In Google's docs you can find an interesting and quick tutorial on how to use it to clean, transform, and process customer data. Then you would just have to sink the transformed data with the BigQuery plugin as you are doing now.