Utf-8到URI百分比编码

Question

I'm trying to convert Unicode code points to percent encoded UTF-8 code units.

The Unicode -> UTF-8 conversion seems to be working correctly as shown by some testing with Hindi and Chinese characters which show up correctly in Notepad++ with UTF-8 encoding, and can be translated back properly.

I thought the percent encoding would be as simple as adding '%' in front of each UTF-8 code unit, but that doesn't quite work. Rather than the expected %E5%84%A3 , I'm seeing %xE5%x84%xA3 (for the unicode U+5123).

What am I doing wrong?

Added code (note that utf8.h belongs to the UTF8-CPP library).

#include <fstream>
#include <iostream>
#include <vector>
#include "utf8.h"

std::string unicode_to_utf8_units(int32_t unicode)
{
    unsigned char u[5] = {0,0,0,0,0};
    unsigned char *iter = u, *limit = utf8::append(unicode, u);
    std::string s;
    for (; iter != limit; ++iter) {
        s.push_back(*iter);
    }
    return s;
}

int main()
{
    std::ofstream ofs("test.txt", std::ios_base::out);
    if (!ofs.good()) {
        std::cout << "ofstream encountered a problem." << std::endl;
        return 1;
    }

    utf8::uint32_t unicode = 0x5123;
    auto s = unicode_to_utf8_units(unicode);
    for (auto &c : s) {
        ofs << "%" << c;
    }

    ofs.close();

    return 0;
}

Answer 1

You actually need to convert byte values to the corresponding ASCII strings, for example:

"é" in UTF-8 is the value { 0xc3, 0xa9 } . Please not that these are bytes, char values in C++.

Each byte needs to be converted to: "%C3" and "%C9" respectively.

The best way to do so is to use sstream :

std::ostringstream out;
std::string utf8str = "\xE5\x84\xA3";

for (int i = 0; i < utf8str.length(); ++i) {
    out << '%' << std::hex << std::uppercase << (int)(unsigned char)utf8str[i];
}

Or in C++11:

for (auto c: utf8str) {
    out << '%' << std::hex << std::uppercase << (int)(unsigned char)c;
}

Please note that the bytes need to be cast to int , because else the << operator will use the litteral binary value. First casting to unsigned char is needed because otherwise, the sign bit will propagate to the int value, causing output of negative values like FFFFFFE5 .