如何正确跳过 unicode (UTF-8) 字符？

Question

I have written a parser that turns out works incorrectly with UTF-8 texts.

The parser is very very simple:

while(pos < end) { 

// find some ASCII char
if (text.at(pos) == '@') {
// Check some conditions and if the syntax is wrong...
if (...)
  createDiagnostic(pos);
} 

pos++;
}

So you can see I am creating a diagnostic at pos . But that pos is wrong if there were some UTF-8 characters (because UTF-8 characters in reality consists of more than one char . How do I correctly skip the UTF-8 chars as if they are one character?

I need this because the diagnostics are sent to UTF-8-aware VSCode.

I tried to read some articles on UTF-8 in C++ but every material I found is huge. And I only need to skip the UTF-8.

Answer 1

If the code point is less than 128, then UTF-8 encodes it as ASCII (No highest bit set). If code point is equal or larger than 128, all the encoded bytes will have the highest bit set. So, this will work:

unsigned char b = <...>; // b is a byte from a utf-8 string
if (b&0x80) {
    // ignore it, as b is part of a >=128 codepoint
} else {
    // use b as an ASCII code
}

Note: if you want to calculate the number of UTF-8 codepoints in a string, then you have to count bytes with:

• !(b&0x80) : this means that the byte is an ASCII character, or
• (b&0xc0)==0xc0 : this means, that the byte is the first byte of a multi-byte UTF8-sequence