How to pass dynamic table names for sink database in Azure Data Factory

Question

I am trying to copy tables from one schema to another with the same Azure SQL db. So far, I have created a lookup pipeline and passed the parameters for the for each loop and copy activity. But my sink dataset is not taking the parameter value I have given under "table option" field rather it is taking the dummy table I chose when creating the sink dataset. Can someone tell how can I pass dynamic table name to a sink dataset?

I have given concat('dest_schema.STG_',@{item().table_name})} in the table option field.

Answer 1

To make the schema and table names dynamic, add Parameters to the Dataset:

Most important - do NOT import a schema. If you already have one defined in the Dataset, clear it. For this Dataset to be dynamic, you don't want improper schemas interfering with the process.

In the Copy activity, provide the values at runtime. These can be hardcoded, variables, parameters, or expressions, so very flexible.

If it's the same database, you can even use the same Dataset for both, just provide different values for the Source and Sink.

WARNING: If you use the "Auto-create table" option, the schema for the new table will define any character field as varchar(8000), which can cause serious performance problems.

MY OPINION:

While you can do this, one of my personal rules is to not cross the database boundary. If the Source and Sink are on the same SQL database, I would try to solve this problem with a Stored Procedure rather than a data factory.