UTF16转换因utfcpp失败

Question

I have this bit of code below that I've written that uses utfcpp to convert from a utf16 encoded file to a utf8 string.

I think I must be using it improperly, because the result isnt changing. The utf8content variable comes out with null characters ( \\0 ) every other character exactly like the uft16 that I put into it.

//get file content
string utf8content;
std::ifstream ifs(path);
vector<unsigned short> utf16line((std::istreambuf_iterator<char>(ifs)), std::istreambuf_iterator<char>());

//convert
if(!utf8::is_valid(utf16line.begin(), utf16line.end())){
    utf8::utf16to8(utf16line.begin(), utf16line.end(), back_inserter(utf8content));
}

I found the location in the library that is doing the append, it treats everything in the first octet the same, whereas my thought is that it should handle 0's differently.

From checked.h here is the append method (line 106). This is called by utf16to8 (line 202). Notice that I added first part of the if, so that it skips the null chars in an attempt to fix the problem.

template <typename octet_iterator>
octet_iterator append(uint32_t cp, octet_iterator result)
{
    if (!utf8::internal::is_code_point_valid(cp))
        throw invalid_code_point(cp);

    if(cp < 0x01)                 //<===I added this line and..
        *(result++);              //<===I added this line
    else if (cp < 0x80)                        // one octet
        *(result++) = static_cast<uint8_t>(cp);
    else if (cp < 0x800) {                // two octets
        *(result++) = static_cast<uint8_t>((cp >> 6)            | 0xc0);
        *(result++) = static_cast<uint8_t>((cp & 0x3f)          | 0x80);
    }
    else if (cp < 0x10000) {              // three octets
        *(result++) = static_cast<uint8_t>((cp >> 12)           | 0xe0);
        *(result++) = static_cast<uint8_t>(((cp >> 6) & 0x3f)   | 0x80);
        *(result++) = static_cast<uint8_t>((cp & 0x3f)          | 0x80);
    }
    else {                                // four octets
        *(result++) = static_cast<uint8_t>((cp >> 18)           | 0xf0);
        *(result++) = static_cast<uint8_t>(((cp >> 12) & 0x3f)  | 0x80);
        *(result++) = static_cast<uint8_t>(((cp >> 6) & 0x3f)   | 0x80);
        *(result++) = static_cast<uint8_t>((cp & 0x3f)          | 0x80);
    }
    return result;
}

I cant imagine that this is the solution however, simply removing the null chars from the string and why wouldnt the library have found this? So clearly I'm doing something wrong.

So, my question is, what is wrong with the way that I'm implementing my utfcpp in the first bit of code? Is there some type conversion that I've done wrong?

My content is a UTF16 encoded xml file. It seems to truncate the results at the first null character.

Answer 1

std::ifstream reads the file in 8bit char units. UTF-16 uses 16bit units instead. So if you want to read the file and fill your vector with proper UTF-16 units, then use std::wifstream instead (or std::basic_ifstream<char16_t> or equivalent if wchar_t is not 16-bit on your platform).

And do no call utf8::is_valid() here. It expects UTF-8 input but you have UTF-16 input instead.

If sizeof(wchar_t) is 2:

std::wifstream ifs(path);
std::istreambuf_iterator<wchar_t> ifs_begin(ifs), ifs_end;
std::wstring utf16content(ifs_begin, ifs_end);
std::string utf8content;

try {
    utf8::utf16to8(utf16content.begin(), utf16content.end(), std::back_inserter(utf8content));
}
catch (const utf8::invalid_utf16 &) {
    // bad UTF-16 data!
}

Otherwise:

// if char16_t is not available, use unit16_t or unsigned short instead

std::basic_ifstream<char16_t> ifs(path);
std::istreambuf_iterator<char16_t> ifs_begin(ifs), ifs_end;
std::basic_string<char16_t> utf16content(ifs_begin, ifs_end);
std::string utf8content;

try {
    utf8::utf16to8(utf16content.begin(), utf16content.end(), std::back_inserter(utf8content));
}
catch (const utf8::invalid_utf16 &) {
    // bad UTF-16 data!
}

Answer 2

The problem is where you're reading the file:

vector<unsigned short> utf16line((std::istreambuf_iterator<char>(ifs)), std::istreambuf_iterator<char>());

This line is taking a char iterator and using it to fill a vector one byte at a time. You're essentially casting each byte instead of reading two bytes at a time.

This is breaking each UTF-16 entity into two pieces, and for much of your input one of those two pieces will be a null byte.