c ++编译器如何将转义序列转换为实际字节？

Question

GCC compiler offer a compiler option (-fexec-charset=option) so you can configure the encoding of your char and string literals, so it convert your string from the source charset ( UTF-8 by default ) to the execution charset.

So I want to know is it this conversion from source charset to execution charset that result the escape sequences to be replaced by their correspendent code point ?

Exmple.

cout << "hello \x60 "; // \x60 replaced by byte 0x60
cout << "hello \n"; // \n replaced by 0xA0

and also in the first example this character \\x60 is encoding independent whereas in the second example, this character '\\n' byte representation is encoding dependent, and also platform dependent (it will change to \\r\\n in windows, and remain \\n on UNIX).

Answer 1

Though you apparently don't quite realize it, you're really asking about two entirely separate conversions.

The first one is converting escape sequences in the compiler. That's pretty straightforward -- when it sees a \\ in (for example) a string, it looks at the next character and produces a single byte of output for the two (or, depending on the exact input, it might be one byte of output from more than two characters of input, such as something like \\001 ).

The conversion from \\n to \\r\\n on Windows is entirely separate -- that happens during output to a stream -- specifically a text-mode stream. That conversion isn't done by the compiler proper at all, but by code in the iostreams library.

In case you really care about the first one, here's some code I wrote years ago that does roughly the same thing as a compiler does (though despite the C++ tag, this code is pure C):

#include <string.h>
#include <stdio.h>
#include "snip_str.h"

char *translate(char *string)
{
      char *here=string;
      size_t len=strlen(string);
      int num;
      int numlen;

      while (NULL!=(here=strchr(here,'\\')))
      {
            numlen=1;
            switch (here[1])
            {
            case '\\':
                  break;

            case 'r':
                  *here = '\r';
                  break;

            case 'n':
                  *here = '\n';
                  break;

            case 't':
                  *here = '\t';
                  break;

            case 'v':
                  *here = '\v';
                  break;

            case 'a':
                  *here = '\a';
                  break;

            case '0':
            case '1':
            case '2':
            case '3':
            case '4':
            case '5':
            case '6':
            case '7':
                  numlen = sscanf(here,"%o",&num);
                  *here = (char)num;
                  break;

            case 'x':
                  numlen = sscanf(here,"%x",&num);
                  *here = (char) num;
                  break;
            }
            num = here - string + numlen;
            here++;
            memmove(here,here+numlen,len-num );
      }
      return string;
}

Answer 2

After searching on the web, I now know the answer to my question. So I will try to explain it for anyone who is wondering about the mechanism of handling escape sequence in c++.

When You write your code on a file you specify your file charset ( Windows-1252 , ISO-8859-1 , UTF-8 , UTF-16 , UTF-16BE , UTF-16LE ...) which will map the characters inside your file to their correspondent code point then get encoded using the charset that you've specified to a stream of bytes to be saved on the hard drive.
When you try to compile your source code file, if you didn't specify what is your file encoding using -finput-charset=option compiler option, the compiler will assume that your file is encoded using UTF-8 . In both cases, the first thing the C PreProcessor (CPP) will do is convert your file into the source charset which is UTF-8.

After the CPP is complete, string and character constants are converted again to the execution charset , by default it matches the source charset UTF-8 but you can change it using -exec-charset=option compiler option. Until now, everything is clear and we didn't talk about escape sequence since they get handled differently.

There is two kinds of escape sequences each get handled differently when the string get converted from the source charset to the execution charset . The first type is octal or hexadecimal escape sequences like \\xA1 or \\45 , the second type is escape sequence that get represented using a backslash followed by a character like \\r or \\n .

Octal and Hexadecimal escape sequence values are independent from the execution charset , which mean they don't get converted from source charset to execution charset , for example \\xA1 has the value A1 regardless of the current execution charset .
The remaining escape sequences values depend on the execution charset , for example '\\n' will get first mapped to the correspondent character in the source charset in this case it is 0A in UTF-8 then converted to execution charset , so for example if the user have set -fexec-charset=UTF-16BE then '\\n' will be 0A in source charset then 00 0A after the source to execution charset conversion.

The Line Feed escape character \\n is even platform dependent, in windows OS the output library will replace \\n=0A with \\r\\n=10 0A , in Unix it will remain \\n=0A . Note that this replacement happen after characters and strings conversion from source charset to execution charset , otherwise we will get different result.