如何使用C ++ 11语言环境设施将UTF-8用作字符串的内部表示？

Question

I'm writing a portable library that deals with files and directories. I want to use UTF-8 for my input (directory paths) and output (file paths). The problem is, Windows gives me a choice between UTF-16-that-used-to-be-UCS-2, and codepages. So I have to convert all my UTF-8 strings to UTF-16, pass them to WinAPI, and convert the results back to UTF-8. C++11 seems to provide <locale> library just for that, except from what I understood, none of the predefined specializations uses UTF-8 as internal (ie. my-side) coding - the closest there is is UTF-16-to-UTF-8, which is the exact opposite of what I want. So here's first question:

1) How to use codecvt thingamajigs to convert my UTF-8 strings to UTF-16 for WinAPI calls, and the UTF-16 results back to UTF-8?

Another problem: I'm also targetting Linux. On Linux, there is a very good support for many different locales - and I don't want to be any different. Hopefully everyone will use UTF-8 on their Linux machines, but there is no strict guarantee of that. So I thought it would be a good idea to extend the above Windows-specific behavior and always do UTF-8-to-system-locale-coding. Except that I don't see there's any way in C++11's <locale> library to get current system encoding! Default std::locale constructor makes specified-by-myself locale, and if I don't do it, it returns classic "C" locale. And there are no other getters I'm aware of. So here's second question:

2) How to detect current system locale? Something in <locale> ? Maybe some standard C library function, or (less portable but okay in this case) something in POSIX API?

Answer 1

The design of these facilities in the standard library assumes that multibyte character encodings (like UTF-8) are used only for external storage (ie byte sequences in files on disk) and that all characters in memory are uniform in size. This is so things like std::basic_string<T>::operator[] can behave in a manner consistent with the performance constraints imposed by the standard. So while you can use files encoded in UTF-8 or some other MBCS (like those for Japanese), your strings in memory should be char , char16_t , char32_t or wchar_t .

This is why you aren't finding a match in the standard library for what you want to do because strings in memory aren't intended to be stored in UTF-8. This is similar to other languages as well, such as Java, where data on disk is interpreted as a stream of bytes and to turn them into strings you need to tell some component the expected character encoding of the byte stream. Some operating systems may stuff a UTF-8 string into argv[] , but this is non-standard. This is the reason that the Unicode enabled entry point for WinMain on Windows provides a NUL terminated pointer to wchar_t and not a char* pointing to a UTF-8 encoded string.

IBM's International Components for Unicode library provides a whole set of components that are complementary to, and design to work with, the C++ standard library. I would look at their code conversion facilities. While the standard defines facilities in <locale> for code conversion, it doesn't guarantee any existence of a code conversion facility to map from UTF-8 to char16_t , char32_t , or wchar_t . If such a thing exists, you'll only get it based on the details of your implementation. The ICU library provides this functionality portably for any C++ implementation. It is well supported and well used and unlikely to have bugs decoding UTF-8 strings into the appropriate wider-than- char string.

Konrad mentioned the UTF-8 Anywhere Manifesto in a comment. This was an interesting read and they point you to the Boost.Nowide library (not officially a part of boost yet) to get solutions to the problems you cite above.

Please note that my answer is simply a description of the way the existing C++ standard library classes like std::basic_string<T> work. It is not advice against UTF-8, Unicode, or anything else. The manifesto cited agrees with me that these things simply don't work this way and if you want to use UTF-8 anywhere, then you need something else.