多余的“隐藏”字符弄乱了SQL中的等于测试

Question

I am doing a database (Oracle) migration validation and I am writing scripts to make sure the target matches the source. My script is returning values that, when you look at them, look equal. However, they are not.

For instance, the target has PREAPPLICANT and the source has PREAPPLICANT . When you look at them in text, they look fine. But when I converted them to hex, it shows 50 52 45 41 50 50 4c 49 43 41 4e 54 for the target and 50 52 45 96 41 50 50 4c 49 43 41 4e 54 for the source. So there is an extra 96 in the hex.

So, my questions are:

1. What is the 96 char?
2. Would you say that the target has incorrect data because it did not bring the char over? I realize this question may be a little subjective, but I'm asking it from the standpoint of "what is this character and how did it get here?"
3. Is there a way to ignore this character in the SQL script so that the equality check passes? (do I want the equality to pass or fail here?)

Answer 1

It looks like you have Windows-1252 character set here. https://en.wikipedia.org/wiki/Windows-1252

Character 96 is an En Dash. This makes sense, as the data was PREAPPLICANT.

One user provided "PREAPPLICANT" and another provided "PRE-APPLICANT" and Windows helpfully converted their proper dash into an en dash.

As such, this doesn't appear to be an error in data, more an error in character sets. You should be able to filter these out without too much effort but then you are changing data. It's kind of like when one person enters "Mr Jones" and another enters "Mr. Jones"--you have to decide how much data massaging you want to do.

As you probably already have done, use the DUMP function to get the byte representation of the data in code of you wish to inspect for weirdness.

Here's some text with plain ASCII:

select dump('Dashes-and "smart quotes"') from dual;

Typ=96 Len=25: 68,97,115,104,101,115,45,97,110,100,32,34,115,109,97,114,116,32,113,117,111,116,101,115,34

Now introduce funny characters:

select dump('Dashes—and “smart quotes”') from dual;

Typ=96 Len=31: 68,97,115,104,101,115,226,128,148,97,110,100,32,226,128,156,115,109,97,114,116,32,113,117,111,116,101,115,226,128,157

In this case, the number of bytes increased because my DB is using UTF8. Numbers outside of the valid range for ASCII stand out and can be inspected further.

Here's another way to see the special characters:

select asciistr('Dashes—and “smart quotes”') from dual;

Dashes\\2014and \\201Csmart quotes\\201D

This one converts non-ASCII characters into backslashed Unicode hex.