在不使用3rd party库的情况下基于C *中基于char *的多平台Unicode处理？

Question

The following are bare minimum examples (I know that eg UNICODE/_UNICODE should be defined) that I've found to work:

Linux:

#include <stdio.h>

int main() {
  char* str = "Rölf";
  printf("%s\n", str);
}

Windows:

#include <stdio.h>
#include <locale.h>

int main() {
  setlocale(LC_ALL, "");
  wchar_t* str = L"Rölf";
  wprintf(L"%s\n", str);
}

Now, I've read that one way of going about it is to basically "just use UTF-8/char everywhere and worry about platform-specific conversion when you do API calls".

And that would be great - have users provide char* as input for my library and "simply" convert that. So I've tried the following snippet based on this example (I've also seen it in variations elsewhere). If this would actually work, it would be amazing. But it doesn't:

  char* str = u8"Rölf";
  int len = mbstowcs(NULL, str, 0) + 1;
  wchar_t wstr[len];
  mbstowcs(wstr, str, len);
  wprintf(L"%s\n", wstr);

I've also stumbled across discussions about console fonts and whatnot being the cause of faulty rendering, so to demonstrate that this is not a console issue - the following doesn't work either (well - the L"" literal does. The converted u8 literal doesn't):

  MessageBoxW(NULL, wstr, L"Rölf", MB_OK);

Am I misunderstanding the conversion process? Is there a way to make to this work? (Without using eg ICU)

Answer 1

The mbstowcs function converts from a string encoded in the current locale's encoding to wchar_t[] , not from UTF-8 (unless that encoding is UTF-8). On post-April-beta-2018 versions of Windows 10 or later, you actually can fix Windows to use UTF-8 as the encoding for plain char[] strings either as a global setting, or presumably by calling _setmbcp(65001) . Older versions of Windows explicitly forbid this however for dubious historical reasons.

Anyway, you second version of the code which you called "Windows" should work on arbitrary systems if not for a bug in MSVC's wprintf that you worked around: they have the meanings of %ls and %s backwards for the wide stdio functions. In standard C, you need %ls to format a wchar_t[] string. But there's actually no reason to use wprintf there at all, and in fact wprintf is highly problematic because you can't mix it with byte-oriented stdio (doing so invokes undefined behavior). So better would be:

#include <stdio.h>
#include <locale.h>

int main() {
  setlocale(LC_ALL, "");
  wchar_t* str = L"Rölf";
  printf("%ls\n", str);
}

and this version should work correctly on Windows and standards-conforming C implementations, since for the byte-oriented printf functions, MSVC doesn't have the meaning of %s and %ls reversed.

If you really want to, you can also use a variant of your third version of the code, but you can't use mbstowcs to convert from UTF-8 to wchar_t . Instead you need to either:

1. Assume wchar_t is Unicode-encoded, and convert from UTF-8 to Unicode codepoints with your own (or a third-party library's) UTF-8 decoder. But this is a bad assumption, because MSVC is also non-conforming in that it uses UTF-16 for wchar_t (C explicitly forbids "multi- wchar_t -characters because the mb/wc APIs are inherently incompatible with them), not Unicode codepoint values (equivalent to UTF-32).

2. Convert from UTF-8 to uchar32_t (UTF-32) with your own (or a third-party library's) UTF-8 decoder, then use c32rtomb to convert to wchar_t[] .

3. Use iconv (standard on POSIX systems; available as a third-party library on Windows) to convert directly from UTF-8 to wchar_t .

---

UTF8 option for Windows 10, version 1803+

Answer 2

Thanks to Barmak Shemirani making me aware of MultiByteToWideChar , I've found a solution to this that is even C99 conform. (Which works on Windows 7 by the way)

Note that setlocale() is only necessary for console output to render correctly. I didn't use it to highlight that it doesn't seem to be needed for GUI-related API calls.

#define UNICODE
#define _UNICODE

#include <stdio.h>
#include <windows.h>
//#include <locale.h>

wchar_t* toWide(char* str) {
  int wchars_num = MultiByteToWideChar(CP_UTF8, 0, str, -1, NULL, 0);

  wchar_t* wstr = (wchar_t*)malloc(sizeof(wchar_t) * wchars_num);
  MultiByteToWideChar(CP_UTF8, 0, str, -1, wstr, wchars_num);

  return wstr;
}

int main() {
  // For output in console to render correctly - as far as the font allows anyway...
  //setlocale(LC_ALL, "");

  // PLATFORM-AGNOSTIC DATA STRUCTURE WITH UTF-8 TEXT
  // (Usually not directly next to the platform-specific API calls...)
  char* str = "Rölf";

  // PLATFORM-SPECIFIC TEXT HANDLING
  wchar_t* wstr = toWide(str);
  printf("%ls\n", wstr);

  MessageBox(NULL, wstr, L"Rölf", MB_OK);
  free(wstr);
}

The way I use it is that I declare a data structure to be filled by my users where all text is char* and assumed to be UTF-8. Then in my library, I use platform-specific UI APIs. And in the case of Windows, doing the above UTF-16 conversion is obviously necessary.