Azure Data Factory v2 pipeline double quotes

Question

My source file has nvarchar and numeric columns.

Numeric column has thousand separator, do identify the value comes with double quotes. When i use the quoteChar ("\\"") in file format the numeric value works fine.

Same time the nvarchar column (Name) has multiple double quotes between the data, if i use the quoteChar the values are split into further more columns based on the number of double quotes.

Is there any fix/solution for this?

Answer 1

Based on the properties in Text format, only one character is allowed. So you can't separate the different data types by different delimiter.

You could try to use | column delimiter if your nvarchar doesn't have | character. Or maybe you have to parse your source file to remove the thousand separator in an Azure Function activity before the transfer. Then it could be identified by copy activity in adf.

Answer 2

ADF parser fails while reading text that is encapsulated with double quotes and also have a comma within the text, like "Hello, World". To make it work, set Escape Character and Quote Character properties value to double quote. This will save the whole text with double quotes into destination.