SSIS - Loop through columns

Question

For some time now, we have been using excel files to store our purchases data. This spreadsheet file has a structure that is uniform. As it has quite a lot of attributes, I am just going to name the most important ones.

In this Excel files we have columns for the products ID, the sizes of each product and the quantities for each composite of product id and size.

The issue is, that we have several stores and each one of them received different quantities per product. So, in each excel, we have a column for each store, in which we place the quantities ordered from each composite.

So, what I want to do is to loop through the columns of each store to add the corresponding quantities of each composite to the specific store.

For example,

INSERT INTO MyTable (product_ID, size_ID, store_ID, quantity)
    VALUES (12345, 34, Mirkwood, 1)

and then, repeat for each column.

I have been trying to find a solution for this, but haven't figure it out just yet. I would really appreciate all the help or tips.

Answer 1

Avoid trying to create some complex, dynamic, looping code in SSIS.

It would be much simpler to split the flow so you have the product_id , size_id and data for one store. Perform your insert for each store using its store_id , which you already know.

Although a looping solution sounds good, and might mean you don't have to update your code when you add a new store, but in SSIS it will be complex to implement and I'm guessing you are not adding stores that frequently to make the effort worthwhile.

Answer 2

You can use unpivot component

The columns checked will be the columns you need to transform into rows. Destination column is the column you will put the data in. Pivot key value column name: name of pivot columns.

After that, you can just insert the rows normally.

Another way of doing it is though sql, I am using t-sql here, but other database should have something similar. First import your raw data into a table. Then using unpivot to do something similar as above.

    CREATE TABLE #pvt (VendorID int, Emp1 int, Emp2 int,
        Emp3 int, Emp4 int, Emp5 int);
    GO
    INSERT INTO #pvt VALUES (1,4,3,5,4,4);
    INSERT INTO #pvt VALUES (2,4,1,5,5,5);
    INSERT INTO #pvt VALUES (3,4,3,5,4,4);
    INSERT INTO #pvt VALUES (4,4,2,5,5,4);
    INSERT INTO #pvt VALUES (5,5,1,5,5,5);
    GO

    select * from #pvt

    --Unpivot the table.
    SELECT VendorID, Employee, Orders – destination table, first column is row group?
    FROM 
       (SELECT VendorID, Emp1, Emp2, Emp3, Emp4, Emp5
       FROM #pvt) p – detail source
    UNPIVOT
       (Orders FOR Employee IN –orders: data, For: column group
          (Emp1, Emp2, Emp3, Emp4, Emp5)
    )AS unpvt;

The only issue with this approach is it can't handle if there is more column (store) being added. But you can make it dynamic: https://www.mssqltips.com/sqlservertip/3002/use-sql-servers-unpivot-operator-to-dynamically-normalize-output/