SQL Server在nvarchar字符串中选择Unicode空字符

Question

I have content that has been imported into our SQL Server 2008 database (using collation SQL_Latin1_General_CP1_CI_AS ) that is contaminated with UNICODE NULLS in nvarchar(128) columns.

The impact is that it blows up our java libraries when they try to export the content in PDF reports and other such manipulations.

I am trying to locate and modify the values in the various tables and columns. I am told by some of our staff that the offending values look like 'usernam e' instead of 'username' .

In trying to find these offending UNICODE NULLS, I've run the following SQL:

SELECT name 
FROM users
WHERE name LIKE '%[^ -~]%' COLLATE Latin1_General_BIN

Returned is the following set:

M
M
M
N
S
S
S
S
ÿþA

I think that these one-letter values might be followed by UNICODE NULLS, but I don't know for sure. the final one certainly looks suspicious as well.

Is there some way of using CONVERT and the hex value -- 0x00 to locate UNICODE NULLS in nvarchar strings?

EDIT #1:

select name, CAST(RIGHT(name,1) AS varbinary(128)) AS RIGHTER_1,
from users
where id=1

returns:

B   0x4200

So, that letter 'B' is a bit funny. There really are UNICODE NULLS here, and the libraries are not architected to handle UNICODE. They're rock solid with LATIN UTF8 chars.

Answer 1

You could use CAST(name AS varbinary(128)) to see the value as hex and examine it.

You could find 'null characters' using a condition name LIKE '%'+CHAR(0)+'%' , however, a valid unicode string could contain zeroes as well, so this is probably not what you need to do.

Are you sure that the problem is not in your libraries, or in PDF generator? It looks like you have unicode strings in the database, but the application is interpreting them as ASCII strings.

Answer 2

Trying to look for null unicode character sequences using varbinary conversions can result in false positives, for example the following unicode in UTF16 LE:

20 00 00 A0

The string is a space followed by a unicode character A0. Both are valid non-null characters. However if you did this:

where charindex (0x0000, cast(UnicodeText as varbinary (max))) > 0

You would get a false positive between the end of the space and the beginning of the next character.

Here's a function that I wrote. Note, it does not perform very well with large text, something I'm working on improving. Possibly a CLR proc would work better. Try this:

    create function dbo.FindNullUnicode
(
    @Input nvarchar(max)
    ,@StartPosition bigint = 1
)
returns bigint
as
begin
    if @StartPosition < 1
        set @StartPosition = 1;

    declare @pos bigint = @StartPosition;
    declare @len bigint = len(@Input);
    declare @singlechar nchar(1);

    while (@pos <= @len)
    begin
        if unicode(SUBSTRING(@input,@pos,1)) = 0 
            return @pos;

        set @pos +=1;
    end;
    return 0;
end

Answer 3

Give that the original post is more than 9 months old, this is, I am sure, too late for the poster. But, per the documentation , the nchar and nvarchar data types are Unicode. They are defined as:

| String data types that are either fixed-length, nchar , or variable-length, nvarchar , Unicode data and use the UNICODE UCS-2 character set.

UCS-2 means each character in the column occupies 2 bytes. If the data is single byte characters, the high-order byte will be 0x00, naturally and every other octet be 0x00.

The original problem was that the consumer was almost certainly expecting ASCII or UTF-8 data rather than UCS-2/UTF-16. Most likely the columns should have been declared as char / varchar rather than nchar / nvarchar . The proper solution would be to do one of the following:

• Alter the table so the columns are the correct data type
• Alter the query to transform the columns using the convert() function, thus: convert(varchar(4000),my_nvarchar_column)
• alter the consumer to properly consume the double-byte characters.