在 C++ 中解释 UTF-8 Unicode 字符串

Question

Currently coding in C++20 using WSL2 Ubuntu, G++.

If I had a .txt file consisting of utf-8 unicode characters:

▄  ▄ ▄▄▄ ▄   ▄   ▄▄▄▄  ▄▄  ▄   ▄ ▄▄▄

1. How can I get the length (number of unicode characters) of this unicode string?

2. How can I read the file content and print out the unicode string?

Answer 1

Assumptions:

• stdout supports UTF-8 (on Windows you can get by with chcp 65001 at the cmd prompt)
• We're counting Unicode code points, not glyphs made up of multiple code points.

UTF-8 encoding consists of start bytes following the bit patterns:

1. 0xxxxxxx (single byte encoding)
2. 110xxxxx (two-byte encoding)
3. 1110xxxx (three-byte encoding)
4. 11110xxx (four-byte encoding)

Follow-on bytes use 10xxxxxx as a bit pattern.

UTF-8 can be read using std::string and the bytes processed accordingly.

Demo code:

#include <iostream>
#include <fstream>
#include <string>

using namespace std;

int main() {
    ifstream f("input.txt");
    string s;
    getline(f,s);
    cout << "string: " << s << endl;
    cout << "length(bytes): " << s.length() << endl;

    int codepoints = 0;
    for(auto b : s) {
        if((b & 0xC0) != 0x80) // not UTF-8 intermediate byte?
            ++codepoints;
    }

    cout << "length(code points): " << codepoints << endl;
}

Output:

string: ▄  ▄ ▄▄▄ ▄   ▄   ▄▄▄▄  ▄▄  ▄   ▄ ▄▄▄
length(bytes): 72
length(code points): 36