Azure Data Factory copy activity with stored procedure

Question

Is there a workaround for the fact that you need to name the first parameter of the stored procedure (the one containing the table type) exactly as the property "tableName" in the input dataaset?

Im using Azure Data Factory V1.

Input dataset (On-premise Oracle source)

{
"name": "DS-ORA-WMS-CDC-DLYTRN",
"properties": {
    "published": false,
    "type": "OracleTable",
    "linkedServiceName": "LS-ORA-WMS-CDC",
    "typeProperties": {
        "tableName": "WMST.DLYTRN"
    },
    "availability": {
        "frequency": "Hour",
        "interval": 1
    },
    "external": true,
    "policy": {}
}}

Output dataset (Azure SQL database)

{
"name": "DS-ASQL-ANALYTICS-DLYTRN",
"properties": {
    "published": false,
    "type": "AzureSqlTable",
    "linkedServiceName": "LS-SQL-ANALYTICS-DB",
    "typeProperties": {
        "tableName": "wms.DLYTRN"
    },
    "availability": {
        "frequency": "Hour",
        "interval": 1
    }
}}

Pipeline

{
"name": "test",
"properties": {
    "description": "test pipeline",
    "activities": [
        {
            "type": "Copy",
            "typeProperties": {
                "source": {
                    "type": "OracleSource",
                    "oracleReaderQuery": "select * from WMST.DLYTRN"
                },
                "sink": {
                    "type": "SqlSink",
                    "sqlWriterStoredProcedureName": "wms.spPersistDlytrn",
                    "storedProcedureParameters": {
                        "srcdc": {
                            "value": "CDC"
                        }
                    },
                    "sqlWriterTableType": "wms.DLYTRNType",
                    "writeBatchSize": 0,
                    "writeBatchTimeout": "00:00:00"
                }
            },
            "inputs": [
                {
                    "name": "DS-ORA-WMS-CDC-DLYTRN"
                }
            ],
            "outputs": [
                {
                    "name": "DS-ASQL-ANALYTICS-DLYTRN"
                }
            ],
            "policy": {
                "timeout": "1.00:00:00",
                "concurrency": 1,
                "retry": 3
            },
            "scheduler": {
                "frequency": "Hour",
                "interval": 1
            },
            "name": "TestWMSCopyWithSproc"
        }
    ],
    "start": "2018-01-04T07:00:00Z",
    "end": "2018-01-08T00:00:00Z",
    "isPaused": false,
    "hubName": "hub",
    "pipelineMode": "Scheduled"
}}

Stored procedure

CREATE PROCEDURE [wms].[spPersistDlytrn]
   @DLYTRNTable [wms].[DLYTRNType] READONLY,
   @srcdc VARCHAR(4)
AS
...
RETURN 0

When running the activity it return the below error message:

Database operation failed on server 'Sink:tcp:someservername.database.windows.net,1433' with SQL Error Number '349'. Error message from database execution : The procedure "spPersistDlytrn" has no parameter named "@wms.DLYTRN"..

As it is not possible to name the stored procedure parameter "wms.DLYTRN" is there a way to exclude the schema prefix?

Answer 1

I solved exactly the same issue recently. Edit code of the dataset your copy activity references as "Source", so its typeProperties section contains table name without a schema prefix. For example:

"typeProperties": {
    "tableName": "DLYTRN"
}

Also, the name of the first parameter of your procedure must match the name of the table so it should be @DLYTRN instead of @DLYTRNTable .

Answer 2

I cannot test this right now, but as it is said here https://docs.microsoft.com/en-us/azure/data-factory/v1/data-factory-invoke-stored-procedure-from-copy-activity#stored-procedure-definition "The name of the first parameter of stored procedure must match the tableName defined in the dataset JSON".

In the example it declares two parameters in the sp:

• @Marketing [dbo].[MarketingType] READONLY
• @stringData varchar(256)

At the dataset it doesnt use the schema prefix, it just says: "tableName": "Marketing", without the schema (try this, as you have the schema in your output dataset definition).

Then at the pipeline, it just gives value for stringData. Also this: "SqlWriterTableType": "MarketingType", see that it doesnt have the schema prefix, and your pipeline definition does have it.

So to sum it up: MarketingType is the actual name of the table and is at the SqlWriterTableType property of the copy activity, while Marketing is the name of the parameter in the stored procedure, and the name of the tablename in the output dataset.

Hope this helps!