如何在iOS中将UTF-8字符串转换为wchars？

Question

I have a Win32 function which I need to port to iOS:

// Loads UTF-8 file and converts to a UTF-16 string

bool LoadUTF8File(char const *filename, wstring &str)
{
    size_t size;
    bool rc = false;
    void *bytes = LoadFile(filename, &size);
    if(bytes != 0)
    {
        int len = MultiByteToWideChar(CP_UTF8, 0, (LPCCH)bytes, size, 0, 0);
        if(len > 0)
        {
            str.resize(len + 1);
            MultiByteToWideChar(CP_UTF8, 0, (LPCCH)bytes, size, &str[0], len);
            str[len] = '\0';
            rc = true;
        }
        delete[] bytes;
    }
    return rc;
}

// LoadFile returns the loaded file as a block of memory
// There is a 3 byte BOM which MultiByteToWideChar seems to ignore
// The text in the file is encoded as UTF-8

I'm using C++ for this, rather than Objective C, and I've been trying to use mbstowcs and _mbstowcs_l. They don't seem to behave in the same way as MultiByteToWideChar. For example, the accented character at the end of the word attaché is not being correctly converted (the Win32 version correctly converts it). Is there a 'UTF-8 to UTF-16' function in the standard libraries somewhere?

Does the Win32 version have a bug in it which I'm not noticing?

The length returned from MultiByteToWideChar is less than the length return from mbstowcs.

Weirdly, in this small test case

    char *p = "attaché";

    wstring str;
    size_t size = strlen(p);
    setlocale(LC_ALL, "");
    int len = mbstowcs(null, p, size);
    if(len > 0)
    {
        str.resize(len + 1);
        mbstowcs(&str[0], p, size);
        str[len] = '\0';
    }
    TRACE(L"%s\n", str.c_str());

    len = MultiByteToWideChar(CP_UTF8, 0, p, size, null, 0);
    if(len > 0)
    {
        str.resize(len + 1);
        MultiByteToWideChar(CP_UTF8, 0, p, size, &str[0], len);
        str[len] = '\0';
    }
    TRACE(L"%s\n", str.c_str());

I get the correct output from mbcstowcs and MultiByteToWideChar erroneously converts the last character into 65533 (REPLACEMENT_CHARACTER). Now I'm confused...

Answer 1

Are you stuck with using C++ for this or is it just the way you choose so far but are open to do it in Objective-C too ?

In Objective-C you can use [yourUTF8String dataUsingEncoding:NSUTF16StringEncoding] to get NSData containing the bytes of the UTF-16 representation of the string.

---

Additional hypothesis: Note that your "é" character that does not get correctly converted in your example may also be explained by the fact that your solution may not take NFD form (or NFC form, either one). This means that if the "é" character is encoded in NFD for as in "the character 'e' with a acute accent" it may not be interpreted correctly whereas the NFC form (as in "the accented e character", ie the pre-composed character directly) it will. Or vice-versa.

That's just one hypothesis, in fact it depends on what result you have instead of the "é" character you expect, but it's worth checking.