Azure Data Factory Copy Activity

Question

I have been working on this for a couple days and cannot get past this error. I have 2 activities in this pipeline. The first activity copies data from an ODBC connection to an Azure database, which is successful. The 2nd activity transfers the data from Azure table to another Azure table and keeps failing.

The error message is: Copy activity met invalid parameters: 'UnknownParameterName', Detailed message: An item with the same key has already been added..

I do not see any invalid parameters or unknown parameter names. I have rewritten this multiple times using their add activity code template and by myself, but do not receive any errors when deploying on when it is running. Below is the JSON pipeline code.

Only the 2nd activity is receiving an error.

Thanks.

Source Data set

{
"name": "AnalyticsDB-SHIPUPS_06shp-01src_AZ-915PM",
"properties": {
    "structure": [
        {
            "name": "UPSD_BOL",
            "type": "String"
        },
        {
            "name": "UPSD_ORDN",
            "type": "String"
        }
    ],
    "published": false,
    "type": "AzureSqlTable",
    "linkedServiceName": "Source-SQLAzure",
    "typeProperties": {},
    "availability": {
        "frequency": "Day",
        "interval": 1,
        "offset": "04:15:00"
    },
    "external": true,
    "policy": {}
}

}

Destination Data set

{
"name": "AnalyticsDB-SHIPUPS_06shp-02dst_AZ-915PM",
"properties": {
    "structure": [
        {
            "name": "SHIP_SYS_TRACK_NUM",
            "type": "String"
        },
        {
            "name": "SHIP_TRACK_NUM",
            "type": "String"
        }
    ],
    "published": false,
    "type": "AzureSqlTable",
    "linkedServiceName": "Destination-Azure-AnalyticsDB",
    "typeProperties": {
        "tableName": "[olcm].[SHIP_Tracking]"
    },
    "availability": {
        "frequency": "Day",
        "interval": 1,
        "offset": "04:15:00"
    },
    "external": false,
    "policy": {}
}

}

Pipeline

{
"name": "SHIPUPS_FC_COPY-915PM",
"properties": {
    "description": "copy shipments ",
    "activities": [
        {
            "type": "Copy",
            "typeProperties": {
                "source": {
                    "type": "RelationalSource",
                    "query": "$$Text.Format('SELECT COMPANY, UPSD_ORDN, UPSD_BOL FROM \"orupsd - UPS interface Dtl\" WHERE COMPANY = \\'01\\'', WindowStart, WindowEnd)"
                },
                "sink": {
                    "type": "SqlSink",
                    "sqlWriterCleanupScript": "$$Text.Format('delete imp_fc.SHIP_UPS_IntDtl_Tracking', WindowStart, WindowEnd)",
                    "writeBatchSize": 0,
                    "writeBatchTimeout": "00:00:00"
                },
                "translator": {
                    "type": "TabularTranslator",
                    "columnMappings": "COMPANY:COMPANY, UPSD_ORDN:UPSD_ORDN, UPSD_BOL:UPSD_BOL"
                }
            },
            "inputs": [
                {
                    "name": "AnalyticsDB-SHIPUPS_03shp-01src_FC-915PM"
                }
            ],
            "outputs": [
                {
                    "name": "AnalyticsDB-SHIPUPS_03shp-02dst_AZ-915PM"
                }
            ],
            "policy": {
                "timeout": "1.00:00:00",
                "concurrency": 1,
                "executionPriorityOrder": "NewestFirst",
                "style": "StartOfInterval",
                "retry": 3,
                "longRetry": 0,
                "longRetryInterval": "00:00:00"
            },
            "scheduler": {
                "frequency": "Day",
                "interval": 1,
                "offset": "04:15:00"
            },
            "name": "915PM-SHIPUPS-fc-copy->[imp_fc]_[SHIP_UPS_IntDtl_Tracking]"
        },
        {
            "type": "Copy",
            "typeProperties": {
                "source": {
                    "type": "SqlSource",
                    "sqlReaderQuery": "$$Text.Format('select distinct ups.UPSD_BOL, ups.UPSD_BOL from imp_fc.SHIP_UPS_IntDtl_Tracking ups LEFT JOIN olcm.SHIP_Tracking st ON ups.UPSD_BOL = st.SHIP_SYS_TRACK_NUM WHERE st.SHIP_SYS_TRACK_NUM IS NULL', WindowStart, WindowEnd)"
                },
                "sink": {
                    "type": "SqlSink",
                    "writeBatchSize": 0,
                    "writeBatchTimeout": "00:00:00"
                },
                "translator": {
                    "type": "TabularTranslator",
                    "columnMappings": "UPSD_BOL:SHIP_SYS_TRACK_NUM, UPSD_BOL:SHIP_TRACK_NUM"
                }
            },
            "inputs": [
                {
                    "name": "AnalyticsDB-SHIPUPS_06shp-01src_AZ-915PM"
                }
            ],
            "outputs": [
                {
                    "name": "AnalyticsDB-SHIPUPS_06shp-02dst_AZ-915PM"
                }
            ],
            "policy": {
                "timeout": "1.00:00:00",
                "concurrency": 1,
                "executionPriorityOrder": "NewestFirst",
                "style": "StartOfInterval",
                "retry": 3,
                "longRetryInterval": "00:00:00"
            },
            "scheduler": {
                "frequency": "Day",
                "interval": 1,
                "offset": "04:15:00"
            },
            "name": "915PM-SHIPUPS-AZ-update->[olcm]_[SHIP_Tracking]"
        }
    ],
    "start": "2017-08-22T03:00:00Z",
    "end": "2099-12-31T08:00:00Z",
    "isPaused": false,
    "hubName": "adf-tm-prod-01_hub",
    "pipelineMode": "Scheduled"
}

}

Answer 1

Have you seen this link ?

They get the same error message and suggest using AzureTableSink instead of SqlSink

"sink": {
                    "type": "AzureTableSink",
                    "writeBatchSize": 0,
                    "writeBatchTimeout": "00:00:00"
                }

It would make sense for you too since your 2nd copy activity is Azure to Azure

Answer 2

It could be a red herring but I'm pretty sure "tableName" is a require entry in the typeProperties for a sqlSource. Yours is missing this for the input dataset. Appreciate you have a join in the sqlReaderQuery so probably best to put a dummy (but real) table name in there. Btw, not clear why you are using $$Text.Format and WindowStart/WindowEnd on your queries if you're not transposing these values into the query; you could just put the query between double quotes.