Azure Data Factory Pipeline Copy Data Error. Importing CSV to SQL Table. An item with the same key has already been added

Question

Ive been working on this all day and I can get it to import a single column from the CSV but the minute there are multiple columns I get this error every time. I dont know if its important but the delimiter is a Pipe (|). In total there are 260 columns and 8.8 million rows I'm trying to import all of the field are nvarcher(255).

{
    "errorCode": "2200",
    "message": "ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property '' is invalid: 'An item with the same key has already been added.'.,Source=,''Type=System.ArgumentException,Message=An item with the same key has already been added.,Source=mscorlib,'",
    "failureType": "UserError",
    "target": "Copy data1",
    "details": []
}

Much thanks to anyone who can shed some light on this issue.

Also found this under output:

{
    "dataRead": 4194304,
    "dataWritten": 0,
    "filesRead": 1,
    "sourcePeakConnections": 10,
    "sinkPeakConnections": 2,
    "rowsRead": 0,
    "rowsCopied": 0,
    "copyDuration": 4,
    "throughput": 1024,
    "errors": [
        {
            "Code": 11402,
            "Message": "ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property '' is invalid: 'An item with the same key has already been added.'.,Source=,''Type=System.ArgumentException,Message=An item with the same key has already been added.,Source=mscorlib,'",
            "EventType": 0,
            "Category": 5,
            "Data": {
                "PropertyName": "UnknownParameterName",
                "InvalidParameter": "An item with the same key has already been added."
            },
            "MsgId": null,
            "ExceptionType": null,
            "Source": null,
            "StackTrace": null,
            "InnerEventInfos": []
        }
    ],
    "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West Europe)",
    "usedDataIntegrationUnits": 4,
    "billingReference": {
        "activityType": "DataMovement",
        "billableDuration": [
            {
                "meterType": "AzureIR",
                "duration": 0.06666666666666667,
                "unit": "DIUHours"
            }
        ]
    },
    "usedParallelCopies": 1,
    "executionDetails": [
        {
            "source": {
                "type": "AzureBlobStorage",
                "region": "West Europe"
            },
            "sink": {
                "type": "AzureSqlDatabase",
                "region": "West Europe"
            },
            "status": "Failed",
            "start": "2020-10-08T22:23:53.9269314Z",
            "duration": 4,
            "usedDataIntegrationUnits": 4,
            "usedParallelCopies": 1,
            "profile": {
                "queue": {
                    "status": "Completed",
                    "duration": 1
                },
                "transfer": {
                    "status": "Completed",
                    "duration": 2,
                    "details": {
                        "listingSource": {
                            "type": "AzureBlobStorage",
                            "workingDuration": 0
                        },
                        "readingFromSource": {
                            "type": "AzureBlobStorage",
                            "workingDuration": 0
                        },
                        "writingToSink": {
                            "type": "AzureSqlDatabase",
                            "workingDuration": 0
                        }
                    }
                }
            },
            "detailedDurations": {
                "queuingDuration": 1,
                "transferDuration": 2
            }
        }
    ],
    "dataConsistencyVerification": {
        "VerificationResult": "NotVerified"
    },
    "durationInQueue": {
        "integrationRuntimeQueue": 0
    }
}

Answer 1

So the issue was meta data. Even though the Copy Data had given the headers unique names on the CSV itself they weren't unique because that's how we receive the data. Just made it skip a row and that fixed it. Thanks to anyone who was looking and solving this.