Azure 数据工厂。 使用存储源目标和查询的控制表将数据从源加载到目标

Question

I am having a brain freeze as I cannot remember how to create the following pipeline, and I know it's quite simple.

I know this works as I created something like this previous, but it was nearly 2 years ago, and I've moved jobs so don't remember the method exactly.

I have created a table that has the following data, | ColumnName | Data | |:----------------|-------:| |SourceSchemaName | dim | |SourceTableName | Table | |TargetSchemaName | dw | |TargetTableName | Table | |ExtractSQL | SELECT Acol1, Acol2 AS Bcol1 FROM dim.Table | |StoredProc | Load_dimTable |

The query has to have the same column names as the target DB to work, I remember that bit. Then I use a look up to get this data from each table that needs the data. The query can also include a parameter if you want to limit this import.

I should be able to use this query to do the copying data from within a foreach. In the copy data, I have the Use Query selected and have @item().ExtractSQL, However when I try this I get an error 'Table is required for Copy Activity'.

I was wondering if anyone else had used this method at all. I found it quite useful, now I can't remember how I got it to work. It's so frustrating.

Many thanks

Answer 1

This was easier than I thought. Though it take me a while to figure it out. But time can be a wonderful thing.

So in the Lookup activity in the Query I am selecting the columns of the table above... which looks rubbish, but it looks fine in the preview.

ColumnName	Data
SourceSchemaName	dim
SourceTableName	Table
TargetSchemaName	dw
TargetTableName	Table
ExtractSQL	SELECT Acol1, Acol2 AS Bcol1 FROM dim.Table
StoredProc	Load_dimTable

Columns

• Source Schema and Table.
• Target Schema and Table.
• SQL example SELECT Acol1, Acol2 AS Bcol1 FROM dim.Table
• Stored Procedure. This copies the data from staging into a 'working' table.

My problem was the source wasn't set up correctly. With the connection. I required two parameters

Do the same in Sink but use the Targets...

Execute Stored Procedure use the link service and add the Stored Procedure name as '@item().LoadSproc'

As long as your control table is set up correctly and the stored procedure is correct. It works fine. I am using a merge in the stored procedure.

Many thanks