Mssql encoding problem with special characters

Question

We are using the node-mssql package to insert into and read out of our azure mssql database. The database is very simple, because it is just used as a key/value cache, where the value can be quite long and also contain special characters.

DB Schema:

create table cache
(
    id       int identity
        constraint cache_pk
            primary key nonclustered,
    cacheKey varchar(500),
    ttl      bigint,
    value    varchar(max)
)
go

create index cache_cacheKey_index
    on cache (cacheKey desc)
go

create index cache_ttl_index
    on cache (ttl)
go

For some reason, when I insert values into "value", some special characters are not treated well.

Dash – example

turns into:

Dash  example

I have seen the same thing happening with the french apostrophe. I also tried to change the collation already, but that did not help. Also tried it by using nvarchar(max) as column type.

This is the insert code (including the sql):

  const result = await conn.request()
                      .input('key', mssql.VarChar, key)
                      .input('value', mssql.Text, value)
                      .input('ttl', mssql.BigInt, cacheEndTime)
                      .query`INSERT INTO cache (cacheKey, value, ttl) VALUES (@key, @value, @ttl)`;

Can you please help with a correct table structure or sql statement to make this work?

Answer 1

I'm not sure if can help you, but have you checked the collation of the table, the database, and the server? The collate have differents levels.

Answer 2

The answer for your question are in one of this items:

• Server collation
• Table collation
• Field collation
• Cast the insert text

For example, if you create nvarchar (I recommend if you have international scenario) field, cast the text insert like N'text to insert'.

It will work;)

Answer 3

I have found the answer. Like @RealSeik and @Larnu already stated it was probably not a problem with the database or the queries themselves, but rather an input problem.

I realized, that node-sass has a type for Unicode text, where they took care of casting it correctly. So instead of mssql.Text I changed it to mssql.NText .

So now the insert command looks as follows:

  const result = await conn.request()
                      .input('key', mssql.VarChar, key)
                      .input('value', mssql.NText, value)
                      .input('ttl', mssql.BigInt, cacheEndTime)
                      .query`INSERT INTO cache (cacheKey, value, ttl) VALUES (@key, @value, @ttl)`;

I have also added collations to my other scripts, for good measure as well. (that alone did not help, but for good measure)

    ALTER DATABASE MyDbName  
    COLLATE Latin1_General_100_CI_AI_SC_UTF8 ;  
    
    create table cache
    (
        id       int identity
            constraint cache_pk
                primary key nonclustered,
        cacheKey varchar(500) COLLATE Latin1_General_100_CI_AI_SC_UTF8,
        ttl      bigint,
        value    varchar(max) COLLATE Latin1_General_100_CI_AI_SC_UTF8
    )
    go
    
    create index cache_cacheKey_index
        on cache (cacheKey desc)
    go
    
    create index cache_ttl_index
        on cache (ttl)
    go