SSIS平面文件导入中的行尾不一致

Question

I have a large, pipe delineated text file with no text qualifiers, and it looks like whatever spit out this file accidentally spit out false "LF" markers in the last column every few hundred rows. The last column is a descriptive column, and It is not text qualified in any way like it should be. file looks similar to this:

id|data|data|data|data|Description[LF]
id|data|data|data|data|Description[LF]
id|data|data|data|data|Description[LF]
id|data|data|data|data|Descr[LF]
iption[LF]
id|data|data|data|data|Description[LF]
Id|data|data|data|data|Description[LF]
id|data|data|data|data|Descripti[LF]
on[LF]
id|data|data|data|data|Description[LF]
id|data|data|data|data|Description[LF]
id|data|data|data|data|Description[LF]
id|data|data|data|data|Description[LF]
id|data|data|data|data|D[LF]
escription[LF]

I'm pretty new to SSIS and SQL in general, Does anyone have any advice on how to fix this?

Answer 1

I did actually find a way to fix it in Notepad++, because I don't know C# and I don't know SSIS well enough..

The ID was 8 Digits long, and followed by 7 Blank spaces. That was absolutely unique to this file.

In notepad++ I used (Find Extended) to search and replace "\\n"(LF) with nothing

then I used the this expression for find:

(\d\d\d\d\d\d\d\d[[:blank:]][[:blank:]][[:blank:]][[:blank:]][[:blank:]][[:blank:]][[:blank:]])

to find all 8 digit numbers with 7 trailing spaces, and for replace, used this:

\r\n\1

to put a [CR][LF] in front of those 8 digit numbers.

Lo and behold it worked! But either way.. My boss contacted the client and is requesting a better file. Now I get kudos, and we get proper data. Thanks for the advice all!

Answer 2

If I had to take a guess, I would say that this is occurring because of how the file is created... you are probably having data that just happens to include certain special characters which are being incorrectly interpreted as a Line Feed.

Check this site to see if the data within your problem lines match any of these encodings. If this is the case then ultimately you have two options available:

1) Create some elaborate and complicated ETL process to detect and correct the file data before you process it. This is inadvisable as it will be a major pain to create and maintain.

2) Try changing the way this file is produced. Most text export wizards will allow you to place quotes (") around text items so that your import process can quickly detect something as a text block as opposed to a series of encoded characters to interpret.