SCD type 2 implementation in Cloud Data Fusion Pipeline

Question

I have requirement to apply SCD2 in my data fusion pipeline and my source data is GCS and target is BigQuery. I have created the pipeline and able to load the data into target BigQery so now I want to implement merge and update process as part of SCD2. Can you please help me as I'm new to Google cloud data fusion.

Answer 1

@phaneendra's approach is a possible solution. Anyhow, Data Fusion is designed to be used in a code-free visual way. If you want to take advantage of its potential, I would recommend you to use Wrangler for your data transformations. In Google's docs you can find an interesting and quick tutorial on how to use it to clean, transform, and process customer data. Then you would just have to sink the transformed data with the BigQuery plugin as you are doing now.