如何将Unicode转义字符转换为utf8？

Question

I saw the other questions about the subject but all of them were missing important details: I want to convert \%2F\%2F\מ\ר\כ\ז to utf8. I understand that you look through the stream for \\u followed by four hex which you convert to bytes. The problems are as follows:

1. I heard that sometimes you look for 4 bytes after and sometimes 6 bytes after, is this correct? If so, then how do you determine which it is? Eg is \%2F 4 or 6 bytes?
2. In the case of \% this maps to one byte instead of two (0x25), why? Is the four hex supposed to represent utf16 which i am supposed to convert to utf8?
3. How do I know whether the text is supposed to be the literal characters \% or the unicode sequence? Does that mean that all backslashes must be escaped in the stream?
4. Lastly, am I being stupid in doing this by hand when I can use iconv to do this for me?

Answer 1

If you have the iconv interfaces at your disposal, you can simply convert the \ģ\ꯍ etc. sequences to an array of bytes 01 23 AB CD ..., replacing any unescaped ASCII characters with a 00 byte followed by the ASCII byte, then run the array through iconv with a conversion descriptor obtained by iconv_open("UTF-8", "UTF-16-BE") .

Of course you can also do it much more efficiently working directly with the input yourself, but that requires reading and understanding the Unicode specification of UTF-16 and UTF-8.

Answer 2

In some conventions (like C++11 string literals), you parse a specific number of hex digits, like four after \\u\u003c/code> and eight after \\U . That may or may not be the convention with the input you provided, but it seems a reasonable guess. Other styles, like C++'s \\x you parse as many hex digits as you can find after the \\x , which means that you have to jump through some hoops if you do want to put a literal hex digit immediately after one of these escaped characters.

Once you have all the values, you need to know what encoding they're in (eg, UTF-16 or UTF-32) and what encoding you want (eg, UTF-8). You then use a function to create a new string in the new encoding. You can write such a function (if you know enough about both encoding formats), or you can use a library. Some operating systems may provide such a function, but you might want to use a third-party library for portability.