为什么C＃Unicode范围覆盖有限范围（最大0xFFFF）？

Question

I'm getting confused about C# UTF8 encoding...

Assuming those "facts" are right:

1. Unicode is the "protocol" which define each character.
2. UTF-8 define the "implementation" - how to store those characters.
3. Unicode define character range from 0x0000 to 0x10FFFF ( source )

According to C# reference , the accepted ranges for each char is 0x0000 to 0xFFFF. I don't understand what about the other character, which above 0xFFFF, and defined in Unicode protocol?

In contrast to C#, when I using Python for writing UTF8 text - it's covering all the expected range (0x0000 to 0x10FFFF). For example:

u"\U00010000"  #WORKING!!!

which isn't working for C#. What's more, when I writing the string u"\\U00010000" (single character) in Python to text file and then read it from C#, this single character document became 2 characters in C#!

# Python (write):
import codecs                        
with codes.open("file.txt", "w+", encoding="utf-8") as f:                        
    f.write(text) # len(text) -> 1

// C# (read): 
string text = File.ReadAllText("file.txt", Encoding.UTF8); // How I read this text from file.
Console.Writeline(text.length); // 2

Why? How to fix?

Answer 1

According to C# reference, the accepted ranges for each char is 0x0000 to 0xFFFF. I don't understand what about the other character, which above 0xFFFF, and defined in Unicode protocol?

Unfortunately, a C#/.NET char does not represent a Unicode character.

A char is a 16-bit value in the range 0x0000 to 0xFFFF which represents one “UTF-16 code unit”. Characters in the ranges U+0000–U+D7FF and U+E000–U+FFFF, are represented by the code unit of the same number so everything's fine there.

The less-often-used other characters, in the range U+010000 to U+10FFFF, are squashed into the remaining space 0xD800–0xDFFF by representing each character as two UTF-16 code units together, so the equivalent of the Python string "\\U00010000" is C# "\?\?" .

Why?

The reason for this craziness is that the Windows NT series itself uses UTF-16LE as the native string encoding, so for interoperability convenience .NET chose the same. WinNT chose that encoding—at the time thought of as UCS-2 and without any of the pesky surrogate code unit pairs—because in the early days Unicode only had characters up to U+FFFF, and the thinking was that was going to be all anyone was going to need.

How to fix?

There isn't really a good fix. Some other languages that were unfortunate enough to have based their string type on UTF-16 code units (Java, JavaScript) are starting to add methods to their strings to do operations on them counting a code point at a time; but there is no such functionality in .NET at present.

Often you don't actually need to consistently need to count/find/split/order/etc strings using proper code point items and indexes. But when you really really do, in .NET, you're in for a bad time. You end up having to re-implement each normally-trivial method by manually walking over each char and check it for being part of a two-char surrogate pair, or converting the string to an array of codepoint ints and back. This isn't a lot of fun, either way.

A more elegant and altogether more practical option is to invent a time machine, so we can send the UTF-8 design back to 1988 and prevent UTF-16 from ever having existed.

Answer 2

Unicode has so-called planes ( wiki ).

As you can see, C#'s char type only supports the first plane, plane 0, the basic multilingual plane .

I know for a fact that C# uses UTF-16 encoding, so I'm a bit surprised to see that it doesn't support code points beyond the first plane in the char datatype. (haven't run into this issue myself...).

This is an artificial restriction in char 's implementation, but one that's understandable. The designers of .NET probably didn't want to tie the abstraction of their own character datatype to the abstraction that Unicode defines, in case that standard would not survive (it already superseded others). This is just my guess of course. It just "uses" UTF-16 for memory representation.

UTF-16 uses a trick to squash code points higher than 0xFFFF into 16 bits, as you can read about here . Technically those code points consist of 2 "characters", the so-called surrogate pair . In that sense it breaks the "one code point = one character" abstraction.

You can definitely get around this by working with string and maybe arrays of char . If you have more specific problems, you can find plenty of information on StackOverflow and elsewhere about working with all of Unicode's code points in .NET.