One file lib to conv utf8 (char*) to wchar_t?

Question

I am using libjson which is awesome. The only problem I have is I need to convert an utf8 string ( char* ) to a wide char string ( wchar_t* ). I googled and tried 3 different libs and they ALL failed (due to missing headers).

I don't need anything fancy. Just a one way conversion. How do I do this?

Answer 1

If you're on windows (which, chances are you are, given your need for wchar_t), use MultiByteToWideChar function (declared in windows.h), as so:

int length = MultiByteToWideChar(CP_UTF8, 0, src, src_length, 0, 0);
wchar_t *output_buffer = new wchar_t [length];
MultiByteToWideChar(CP_UTF8, 0, src, src_length, output_buffer, length);

Alternatively, if all you're looking for is a literal multibyte representation of your UTF8 (which is improbable, but possible), use the following (stdlib.h):

wchar_t * output_buffer = new wchar_t [1024];
int length = mbstowcs(output_buffer, src, 1024);
if(length > 1024){
    delete[] output_buffer;
    output_buffer = new wchar_t[length+1];
    mbstowcs(output_buffer, src, length);
}

Hope this helps.

Answer 2

the below successfully enables CreateDirectoryW() to write to C:\\Users\\ПетрКарасев , basically an easier-to-understand wrapper around the MultiByteTyoWideChar mentioned by someone earlier.

std::wstring utf16_from_utf8(const std::string & utf8)
{
    // Special case of empty input string
if (utf8.empty())
    return std::wstring();

// Шаг 1, Get length (in wchar_t's) of resulting UTF-16 string
const int utf16_length = ::MultiByteToWideChar(
    CP_UTF8,            // convert from UTF-8
    0,                  // default flags
    utf8.data(),        // source UTF-8 string
    utf8.length(),      // length (in chars) of source UTF-8 string
    NULL,               // unused - no conversion done in this step
    0                   // request size of destination buffer, in wchar_t's
    );
if (utf16_length == 0)
{
    // Error
    DWORD error = ::GetLastError();
    throw ;
}


// // Шаг 2, Allocate properly sized destination buffer for UTF-16 string
std::wstring utf16;
utf16.resize(utf16_length);

// // Шаг 3, Do the actual conversion from UTF-8 to UTF-16
if ( ! ::MultiByteToWideChar(
    CP_UTF8,            // convert from UTF-8
    0,                  // default flags
    utf8.data(),        // source UTF-8 string
    utf8.length(),      // length (in chars) of source UTF-8 string
    &utf16[0],          // destination buffer
    utf16.length()      // size of destination buffer, in wchar_t's
    ) )
{
    // не работает сука ... 
    DWORD error = ::GetLastError();
    throw;
}

return utf16; // ура!
}

Answer 3

Here is a piece of code i wrote. It seems to work well enough. It returns 0 on utf8 error or when the value is > FFFF (which cant be held by a wchar_t)

#include <string>
using namespace std;
wchar_t* utf8_to_wchar(const char*utf8){
    wstring sz;
    wchar_t c;
    auto p=utf8;
    while(*p!=0){
        auto v=(*p);
        if(v>=0){
            c = v;
            sz+=c;
            ++p;
            continue;
        }
        int shiftCount=0;
        if((v&0xE0) == 0xC0){
            shiftCount=1;
            c = v&0x1F;
        }
        else if((v&0xF0) == 0xE0){
            shiftCount=2;
            c = v&0xF;
        }
        else
            return 0;
        ++p;
        while(shiftCount){
            v = *p;
            ++p;
            if((v&0xC0) != 0x80) return 0;
            c<<=6;
            c |= (v&0x3F);
            --shiftCount;
        }
        sz+=c;
    }
    return (wchar_t*)sz.c_str();
}

Answer 4

The following (untested) code shows how to convert a multibyte string in your current locale into a wide string. So if your current locale is UTF-8, then this will suit your needs.

const char * inputStr = ... // your UTF-8 input
size_t maxSize = strlen(inputStr) + 1;
wchar_t * outputWStr = new wchar_t[maxSize];
size_t result = mbstowcs(outputWStr, inputStr, maxSize);
if (result == -1) {
    cerr << "Invalid multibyte characters in input";
}

You can use setlocale() to set your locale.