Azure 数据工厂：在通过管道从 csv 导入 sql server 期间发生错误和意外的数据类型转换

Question

I am trying to load data from a csv to a sql server database using an Azure pipeline copy data operator. During the import data is converted to other types.

in the Source preview in the pipeline I see the following

1- the value "0044" is converted to 44

2- the value 2020000000000000 is converted to 2E+16

3- the value 5.2 is converted to February 5th

4- the value 9.78 is converted to September 1978

so far i could not find a solution for 0044,

I the other cases here is what I did:

for 2 I enclosed the number 2020000000000000 in "" then it worked, though for some reason I get it enclosed in four " like so: ""2020000000000000"" for 3 and 4 I replaced the dot for a comma and then it worked.

But I would like to be able to tell the import utility to treat everything just as string and do the conversions in the database.

how can I achive this?

the code shows following for one of the columns in 3 and 4:

(

       "source":(

              "name": "Amount"

              "type": "String"

       )

       "sink":(

              "name": "Amount"

              "type": "String"

       )

)

Best Regards,

Answer 1

All the default data type in csv is String.

For Azure SQL database/SQL Server, we can not store data '0044' as int data type. You need convert '0044' as String:

We could using select convert to 44 to '0044':

select right('0000'+ltrim([a]),4) new_a, b from test12

When we copy data from csv file, you need think about if the data in csv file is valid data type in Azure SQL database/SQL Server. For example, the data '2020000000000000' is out of int length.

It's very important to design the sink table. So the suggestions is that you first create the sink table in you Azure SQL database with the suitable data type for every column, then set the column mappings in Copy active manually:

Mapping settings:

Pipeline running:

Data check in SQL database:

Update:

The issue is solved by Ramiro Kollmannsperger himself now:

"my sink table in the database has only nvarchar columns. I did this so after a lot of headaches with datatypes and length. I decided that it is easier for me to just do the conversions from nvarchar in the database into a staging table. What helped in the end was to do the schema import in the source Dataset where the csv is read. There is a tab "connection" and next to it another tab "schema" where you can import the schema. After doing this it worked."

Hope this helps.

Answer 2

my sink table in the database has only nvarchar columns. I did this so after a lot of headaches with datatypes and length.

I decided that it is easier for me to just do the conversions from nvarchar in the database into a staging table.

What helped in the end was to do the schema import in the source Dataset where the csv is read. There is a tab "connection" and next to it another tab "schema" where you can import the schema. After doing this it worked.