Loading Fact table with SCD2 from Lookup dimension inSSIS

Question

I am having trouble loading a dimension with the Datawarehouse Surrogate Key in the fact table for instance the SCD Dimension I am looking up has 1002 rows, 2 rows which have had changes of SCD2 and the rest as SCD1. When I use the Lookup transformation in SSIS to being back both, it only brings up 1000 rows when it should bring 1002. I am looking up the BusinessKey and bringing back the DWH surrogate key. The problem lies is because I am using the Business Key as a lookup join to bring the DWH Surrogate key, I am certain the Lookup tranformation removes the duplicate Business Key as there are going to be 4 rows which are duplicated.Is there any way around this? Or am i doing it completely wrong

Answer 1

A lookup in SSIS will match only once, the first match, even if there are 2 occurences. Altering this behavior of the lookup in SSIS is not possible.

Before loading your fact table in SSIS, you can prepare the data at SQL level, for example with a view. In the query you can lookup (by joining to dimensions) your values by business key and an extra criterium which could be between start and end date in order to get the right surrogate key.