如何在用UTF8编码的C中打印unicode字符？

Question

I am trying to print magnifying glass ( http://www.fileformat.info/info/unicode/char/1f50e/index.htm ), and I get this error:

[niko@dev1 ncurses]$ gcc -o utf8 -std=c99 $(ncursesw5-config --cflags --libs) utf8.c 
utf8.c: In function ‘main’:
utf8.c:12:10: error: \ud83d is not a valid universal character
   printw("\ud83ddd0e\n");         // escaped Unicode 
          ^
[niko@dev1 ncurses]$ cat utf8.c
#include <locale.h>
#include <curses.h>
#include <stdlib.h>


int main (int argc, char *argv[])
{
  setlocale(LC_ALL, "");

  initscr();

  printw("\ud83ddd0e\n");         // escaped Unicode 

  getch();
  endwin();

  return EXIT_SUCCESS;
}

What is the problem here? For, example, if I have a decimal number of encoding, which for magnifying glass is 55357 , how would I print it in printf to ncurses screen? (without using wchar_t because it wastes a lot of memory)

Answer 1

The information on fileformat.info is wrong. The escapes on the page are \?\? . This is an UTF-16 surrogate pair as used on Java, but it does not work on C, as GCC seems to require that one \\u\u003c/code> escape represent one Unicode codepoint, which half of the surrogate escape is not.

You should instead use \\U (uppercase) with 8 hexadecimal digits, so U+1F50E becomes \\U0001F50E . This escaped character is output correctly with printf .

---

PS: if instead of magnifying glass you see something like ~_~T~N , make sure that you've called the setlocale and actually linked against -lncursesw , failure to do either will mean that garbage will be printed instead.

Answer 2

You should not encode your string in UTF-16 (the \\ud8..\\udd.. ), but in UTF-8. To convert it, run this command:

perl -e 'print pack("H*","d83ddd0e")' | iconv -f UTF-16 -t UTF-32 | hexdump -C

Then, you can see that your character is U+0001F50E. To insert this character back in your C code, use the \\U sequence, with a capital U.

"\U0001F50E"

By the way, your number 55357 is not the magnifying glass (U+1F50E), but only the first half of the magnifying glass encoded in UTF-16.

Answer 3

Some clarification is needed because OP asked more than one question:

• What is the problem here?

  Antti Haapala answered the important part, which dealt with the improperly represented character.

• For, example, if I have a decimal number of encoding, which for magnifying glass is 55357 , how would I print it in printf to ncurses screen? (without using wchar_t because it wastes a lot of memory)

  This was unanswered. The comment about wasting memory overlooks the fact that ncurses (ie, ncursesw ) will store all of that information in complex characters , which use even more memory than wide characters ( wchar_t ).

printw is similar to printf , but not identical. To see this, the printw manual page says

The printw , wprintw , mvprintw and mvwprintw routines are analogous to printf [see printf(3)]. In effect, the string that would be output by printf is output instead as though waddstr were used on the given window.

To understand what analogous means, a dictionary might help (part of its meaning is "similar", but those are not synonymous). But following the link to the waddstr manual page :

These functions write the (null-terminated) character string str on the given window. It is similar to calling waddch once for each character in the string.

Again, "similar" offers no guarantee that the behavior is identical. The waddch manual page gives more information. Among other things, it tells what translations it will do for control- and nonprinting-characters. Also (the point) is that waddch in ncurses accepts a multibyte (read: "UTF-8") string and will display that if the locale and terminal support that. That's different from X/Open Curses, as discussed in the Character Set subsection of the manual page's PORTABILITY section.

Those \\u\u003c/code> escapes tell the gcc to pass a UTF-8 string, which happens to work with ncurses. The people concerned with standards will equivocate on whether it's guaranteed to work with printf , but let's not get into that swamp.

There is, by the way, no equivalent of printw which uses wchar_t arrays.

Answer 4

You can use putwchar (see http://www.cplusplus.com/reference/cwchar/putwchar/ ) to print a wchar, but I don't believe it works for UTF-16 surrogate pairs.

In any case, printing unicode text to the terminal is always undefined behavior. On unix systems, most terminals emulate the VT-100, and are only guaranteed to support 7-bit ASCII text. (this why the isprint function exists).

Your best option is to use a library like freetype2 or cairo+pango to render text to a surface or pixmap in a graphical application.