Rank/Rownumber function in SSIS Dataflow

Question

In my dataflow, after some lookups I would get duplicate customer records(They are not exact duplicates only the customer ID is the same), based on some attributes of the customer like city, location. I need to choose one record among them.

How I can achieve this in SSIS dataflow

Here is the sample data:

;with cust (CustomerID,Cutomer_Name,score)
as 
(Select 1 as CustomerID, 'abd' as Cutomer_Name, 100 as Score
union 
select 1,'abd',null
union select 1,'abd',20
)  

select * from cust   

From here I need to choose the record the with lowest score and send only that row to the final table.

It's easy to achieve with Rownum function in SQL, but this case occurs during the dataflow in SSIS

Answer 1

Do the source's data access mode on an SQL command.

Answer 2

Use a MultiCast to split it into two Outputs - say Output1 and Output2. One of the outputs connect to a Aggregate transformation and Group by CustomerId and do a Minimum of Score. Now connect back the output of the Aggregate transform to Output2 use a Merge Join in the mapping map Output2.CustomerId = Aggregate Transform.Score and Output2.CustomerId = Aggregate Transform.Score. This would do the trick, but if you have multiple customerIds with the same score then you might need a Sort after this step to remove duplicates. Hope this helps.

Answer 3

This is the solution which helped me to solve my issue

http://paultebraak.wordpress.com/2013/02/25/rank-partitioning-in-etl-using-ssis/