Azure 数据工厂管道复制数据错误。 将 CSV 导入 SQL 表。 已添加具有相同键的项目

Question

我一整天都在研究这个，我可以让它从 CSV 导入单个列，但是当有多个列时，我每次都会收到这个错误。 我不知道它是否重要，但分隔符是管道 (|)。 总共有 260 列和 880 万行我试图导入所有字段都是 nvarcher(255)。

{
    "errorCode": "2200",
    "message": "ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property '' is invalid: 'An item with the same key has already been added.'.,Source=,''Type=System.ArgumentException,Message=An item with the same key has already been added.,Source=mscorlib,'",
    "failureType": "UserError",
    "target": "Copy data1",
    "details": []
}

非常感谢任何可以阐明这个问题的人。

在输出下也发现了这个：

{
    "dataRead": 4194304,
    "dataWritten": 0,
    "filesRead": 1,
    "sourcePeakConnections": 10,
    "sinkPeakConnections": 2,
    "rowsRead": 0,
    "rowsCopied": 0,
    "copyDuration": 4,
    "throughput": 1024,
    "errors": [
        {
            "Code": 11402,
            "Message": "ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property '' is invalid: 'An item with the same key has already been added.'.,Source=,''Type=System.ArgumentException,Message=An item with the same key has already been added.,Source=mscorlib,'",
            "EventType": 0,
            "Category": 5,
            "Data": {
                "PropertyName": "UnknownParameterName",
                "InvalidParameter": "An item with the same key has already been added."
            },
            "MsgId": null,
            "ExceptionType": null,
            "Source": null,
            "StackTrace": null,
            "InnerEventInfos": []
        }
    ],
    "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West Europe)",
    "usedDataIntegrationUnits": 4,
    "billingReference": {
        "activityType": "DataMovement",
        "billableDuration": [
            {
                "meterType": "AzureIR",
                "duration": 0.06666666666666667,
                "unit": "DIUHours"
            }
        ]
    },
    "usedParallelCopies": 1,
    "executionDetails": [
        {
            "source": {
                "type": "AzureBlobStorage",
                "region": "West Europe"
            },
            "sink": {
                "type": "AzureSqlDatabase",
                "region": "West Europe"
            },
            "status": "Failed",
            "start": "2020-10-08T22:23:53.9269314Z",
            "duration": 4,
            "usedDataIntegrationUnits": 4,
            "usedParallelCopies": 1,
            "profile": {
                "queue": {
                    "status": "Completed",
                    "duration": 1
                },
                "transfer": {
                    "status": "Completed",
                    "duration": 2,
                    "details": {
                        "listingSource": {
                            "type": "AzureBlobStorage",
                            "workingDuration": 0
                        },
                        "readingFromSource": {
                            "type": "AzureBlobStorage",
                            "workingDuration": 0
                        },
                        "writingToSink": {
                            "type": "AzureSqlDatabase",
                            "workingDuration": 0
                        }
                    }
                }
            },
            "detailedDurations": {
                "queuingDuration": 1,
                "transferDuration": 2
            }
        }
    ],
    "dataConsistencyVerification": {
        "VerificationResult": "NotVerified"
    },
    "durationInQueue": {
        "integrationRuntimeQueue": 0
    }
}

Answer 1

所以问题是元数据。 即使复制数据在 CSV 本身上给了标题唯一的名称，它们也不是唯一的，因为这就是我们接收数据的方式。 只是让它跳过一行并修复它。 感谢任何正在寻找和解决这个问题的人。