java utf8编码 - char，字符串类型

Question

public class UTF8 {
    public static void main(String[] args){
        String s = "ﾖ"; //0xFF6E
        System.out.println(s.getBytes().length);//length of the string
        System.out.println(s.charAt(0));//first character in the string
    }
}

output:

3
ﾖ

Please help me understand this. Trying to understand how utf8 encoding works in java. As per java doc definition of char char: The char data type is a single 16-bit Unicode character.

Does it mean char type in java can only support those unicode characters that can be represented with 2 bytes and not more than that?

In the above program, the no of bytes allocated for that string is 3 but in the third line which returns first character( 2 bytes in java) can hold a character which is 3 bytes long? really confused here?

Any good references regarding this concept in java/ general would be really appreciated.

Answer 1

Nothing in your code example is directly using UTF-8. Java strings are encoded in memory using UTF-16 instead. Unicode codepoints that do not fit in a single 16-bit char will be encoded using a 2-char pair known as a surrogate pair.

If you do not pass a parameter value to String.getBytes() , it returns a byte array that has the String contents encoded using the underlying OS's default charset. If you want to ensure a UTF-8 encoded array then you need to use getBytes("UTF-8") instead.

Calling String.charAt() returns an original UTF-16 encoded char from the String's in-memory storage only.

So in your example, the Unicode character ｮ is stored in the String in-memory storage using two bytes that are UTF-16 encoded ( 0x6E 0xFF or 0xFF 0x6E depending on endian), but is stored in the byte array from getBytes() using three bytes that are encoded using whatever the OS default charset is.

In UTF-8, that particular Unicode character happens to use 3 bytes as well ( 0xEF 0xBD 0xAE ).

Answer 2

String.getBytes() returns the bytes using the platform's default character encoding which does not necessary match internal representation.

You're best of never using this method in most cases, because in most cases it does not make sense to rely on platform's default encoding. Use String.getBytes(String charsetName) instead and explicit specify the character set that should be used for encoding your String into bytes.

Answer 3

UTF-8 is a variable length encoding, that uses only one byte for ASCII chars (values between 0 and 127), and two, three (or even more) bytes for other unicode symbols.

This is because the higher bit of the byte is used to tell "this is a multi byte sequence", so one bit on 8 is not used to actually represent "real" data (the char code) but to mark the byte.

So, despite Java using 2 bytes in ram for each char, when chars are "serialized" using UTF-8, they may produce one, two or three bytes in the resulting byte array, that's how the UTF-8 encoding works.

Answer 4

这就是Java代表字符的方式 。