使用 ADF 将 CSV 个文件导入 SQL 服务器

Question

I've a CSV file stored in a blob storage and want to upsert the records in a on-premise SQL server. Currently I'm getting errors on inserting date fields. In the CSV file those date fields are typed as a date field. In my SQL table they are marked as DateTime2 fields. In the CSV file I have about 1000 records, 500 of them are seen as correct and 500 as incorrect.

When I check the file I really don't see a difference in the date fields. There is no time hidden or whatever which could actually cause my pipeline to fail.

What would be a good way to handle those fields? In the current situation, as a work around so that the business could continue, I've inserted all the fields as nvarchar(500) fields. So the imports works, but it's not ideal...

-- update

I've uploaded an image of the file which I'm importing, removed the fields where I'm not getting errors on. As you can see in the csv file the cells where there is some data are marked as Date fields.

Excel example

In ADF all fields are identified as a 'string' field, so maybe that's the problem. Never the less I can't change the schema in the import settings.

ADF identifying them as string

In SQL they are set to DateTime2 (setting it to dateTime or date gave the same error).

Sink field schema

Here the mapping itself:

Source and Sink mapping

Eventually the error that I'm getting, is saying that it can't insert a string value into a datetime field.

{
"errorCode": "2200",
"message": "ErrorCode=TypeConversionFailure,Er is een uitzondering opgetreden bij het converteren van de waarde 29-8-2017 voor de kolomnaam TT_STARTDATE van het type String (Precision: , Scale: ) naar type DateTime (Precision: 255, Scale: 7). Aanvullende informatie: String was not recognized as a valid DateTime.",
"failureType": "UserError",
"target": "EMP",
"details": [
    {
        "errorCode": 0,
        "message": "'Type=System.FormatException,Message=String was not recognized as a valid DateTime.,Source=mscorlib,'",
        "details": []
    }
]}

Thanks in advance!

Answer 1

Well i think it is the fomat of the date which is different in excel and what SQL is expecting is creating the error. I think in Excel you are providing dd-MM-YYYY, may be your SQL is expecting MM-DD-YYYY. I suggest please check this on the SQL side by inserting a record from TSQL using SSMS or other IDE. Once you are sure you can change the format in EXCEL accordingly.