如何判断一个字符串是否包含 Javascript 中的多字节字符？

Question

Is it possible in Javascript to detect if a string contains multibyte characters? If so, is it possible to tell which ones?

The problem I'm running into is this (apologies if the Unicode char doesn't show up right for you)

s = "𝌆";

alert(s.length);    // '2'
alert(s.charAt(0)); // '��'
alert(s.charAt(1)); // '��'

---

Edit for a bit of clarity here (I hope) . As I understand it now , all strings in Javascript are represented as a series of UTF-16 code points, which means that regular characters actually take up 2 bytes (16 bits), so my usage of "multibyte" in the title was a bit off. Some characters do not fall in the Basic Multilingual Plane (BMP), such as the string in the example above, and so they take up two code points (32 bits). That is the question I was asking. I'm also not editing the original title, since to someone who doesn't know much about this stuff (and hence would be searching SO for info about it), "multibyte" would make sense.

Answer 1

JavaScript strings are UCS-2 encoded but can represent Unicode code points outside the Basic Multilingual Pane ( U+0000 - U+D7FF and U+E000 - U+FFFF ) using two 16 bit numbers (a UTF-16 surrogate pair ), the first of which must be in the range U+D800 - U+DFFF .

Based on this, it's easy to detect whether a string contains any characters that lie outside the Basic Multilingual Plane (which is what I think you're asking: you want to be able to identify whether a string contains any characters that lie outside the range of code points that JavaScript represents as a single character):

function containsSurrogatePair(str) {
    return /[\uD800-\uDFFF]/.test(str);
}

alert( containsSurrogatePair("foo") ); // false
alert( containsSurrogatePair("f𝌆") ); // true

Working out precisely which code points are contained in your string is a little harder and requires a UTF-16 decoder. The following will convert a string into an array of Unicode code points:

var getStringCodePoints = (function() {
    function surrogatePairToCodePoint(charCode1, charCode2) {
        return ((charCode1 & 0x3FF) << 10) + (charCode2 & 0x3FF) + 0x10000;
    }

    // Read string in character by character and create an array of code points
    return function(str) {
        var codePoints = [], i = 0, charCode;
        while (i < str.length) {
            charCode = str.charCodeAt(i);
            if ((charCode & 0xF800) == 0xD800) {
                codePoints.push(surrogatePairToCodePoint(charCode, str.charCodeAt(++i)));
            } else {
                codePoints.push(charCode);
            }
            ++i;
        }
        return codePoints;
    }
})();

alert( getStringCodePoints("f𝌆").join(",") ); // 102,119558

Answer 2

Using more modern Javascript syntax (Chrome 46+):

const isMultiByte = string =>
  [...string].some(c => c.codePointAt(0) > 255)

Examples:

isMultiByte("hi") -> false
isMultiByte("hiÿ") -> false // char code 255, small letter y with diaeresis
isMultiByte("こ") -> true

To find the multi-byte characters, change .some to .filter :

const getMultiByteChars = string =>
  [...string].filter(c => c.codePointAt(0) > 255)

Example:

getMultiByteChars("こydwdこ") -> ['こ', 'こ']

If you want to eliminate duplicates:

const getUniqueMultiByteChars = string =>
  [...string]
    .filter(c => c.codePointAt(0) > 255)
    .reduce((uniq, c) => (
      uniq.includes(c) ? uniq : [...uniq, c]
    ), [])

For positions of multi-byte characters:

const getMultiByteCharsPos = string =>
  [...string].reduce((idxs, c, idx) => (
    c.codePointAt(0) > 255 ? [...idxs, idx] : idxs
  ), [])

Example:

getMultiByteCharsPos("こydwdこ") -> [0, 5]

Note: This doesn't work in IE, no String.CodePointAt(n) . MS has officially EOL'ed Internet Explorer at the time of posting.