Azure 数据工厂架构映射不适用于 SQL 接收器

Question

I have a simple pipeline that loads data from a csv file to an Azure SQL db.

I have added a data flow where I have ensured all schema matches the SQL table. I have a specific field which contains numbers with leading zeros. The data type in the source - projection is set to string. The field is mapped to the SQL sink showing as string data-type. The field in SQL has nvarchar(50) data-type.

Once the pipeline is run, all the leading zeros are lost and the field appears to be treated as decimal:

Original data: 0012345
Inserted data: 12345.0

The CSV data shown in the data preview is showing correctly, however for some reason it loses its formatting during insert.

Any ideas how I can get it to insert correctly?

Answer 1

I had repro'd in my lab and was able to load as expected. Please see the below repro details.

Source file (CSV file):

Sink table (SQL table):

ADF:

1. Connect the data flow source to the CSV source file. As my file is in text format, all the source columns in the projection are in a string.

Source data preview:

2. Connect sink to Azure SQL database to load the data to the destination table.

3. Data in Azure SQL database table.

Note : You can all add derived columns before sink to convert the value to string as the sink data type is a string.

Answer 2

Thank you very much for your response.

As per your post the DF dataflow appears to be working correctly. I have finally discovered an issue with the transformation - I have an Azure batch service which runs a python script, which does a basic transformation and saves the output to a csv file.

Interestingly, when I preview the data in the dataflow, it looks as expected. However, the values stored in SQL are not.

For the sake of others having a similar issue, my existing python script used to convert a 'float' datatype column to string-type. Upon conversion, it used to retain 1 decimal number but as all of my numbers are integers, they were ending up with.0.

The solution was to convert values to integer and then to string:

df['col_name'] = df['col_name'].astype('Int64').astype('str')