UTF-8 to Unicode conversion

Question

I am having problems with converting UTF-8 to Unicode.

Below is the code:

int charset_convert( char * string, char * to_string,char* charset_from, char* charset_to)
{
    char *from_buf, *to_buf, *pointer;
    size_t inbytesleft, outbytesleft, ret;
    size_t TotalLen;
    iconv_t cd;

    if (!charset_from || !charset_to || !string) /* sanity check */
        return -1;

    if (strlen(string) < 1)
        return 0; /* we are done, nothing to convert */

    cd = iconv_open(charset_to, charset_from);
    /* Did I succeed in getting a conversion descriptor ? */
    if (cd == (iconv_t)(-1)) {
        /* I guess not */
        printf("Failed to convert string from %s to %s ",
              charset_from, charset_to);
        return -1;
    }
    from_buf = string;
    inbytesleft = strlen(string);
    /* allocate max sized buffer, 
       assuming target encoding may be 4 byte unicode */
    outbytesleft = inbytesleft *4 ;
    pointer = to_buf = (char *)malloc(outbytesleft);
    memset(to_buf,0,outbytesleft);
    memset(pointer,0,outbytesleft);

        ret = iconv(cd, &from_buf, &inbytesleft, &pointer, &outbytesleft);ing
    memcpy(to_string,to_buf,(pointer-to_buf);
}

main():

int main()
{    
    char  UTF []= {'A', 'B'};
    char  Unicode[1024]= {0};
    char* ptr;
    int x=0;
    iconv_t cd;

    charset_convert(UTF,Unicode,"UTF-8","UNICODE");

    ptr = Unicode;

    while(*ptr != '\0')
    {   
        printf("Unicode %x \n",*ptr);
        ptr++;
    }
    return 0;
}

It should give A and B but i am getting:

ffffffff
fffffffe
41 

Thanks, Sandeep

Answer 1

It looks like you are getting UTF-16 out in a little endian format:

ff fe 41 00 ...

Which is U+FEFF (ZWNBSP aka byte order mark), U+0041 (latin capital letter A), ...

You then stop printing because your while loop has terminated on the first null byte. The following bytes should be: 42 00 .

You should either return a length from your function or make sure that the output is terminated with a null character (U+0000) and loop until you find this.

Answer 2

UTF-8 is Unicode.

You do not need to covert unless you need some other type of Unicode encoding like UTF-16, or UTF-32

Answer 3

UTF is not Unicode. UTF is an encoding of the integers in the Unicode standard. The question, as is, makes no sense. If you mean you want to convert from (any) UTF to the unicode code point (ie the integer that stands for an assigned code point, roughly a character), then you need to do a bit of reading, but it involves bit-shifting for the values of the 1, 2, 3 or 4 bytes in UTF-8 byte sequence (see Wikipedia , while Markus Kuhn's text is also excellent)

Answer 4

Unless I am missing something as nobody has pointed it out yet, "UNICODE" isn't a valid encoding name in libiconv as it is the name of a family of encodings.

http://www.gnu.org/software/libiconv/

(edit) Actually iconv -l shows UNICODE as a listed entry but no details, in the source code its listed in the notes as an alias for UNICODE-LITTLE but in the subnotes it mentions:

 * UNICODE (big endian), UNICODEFEFF (little endian)
   We DON'T implement these because they are stupid and not standardized.

In the aliases header files UNICODELITTLE (no hyphen) resolves as follows:

lib/aliases.gperf:UNICODELITTLE, ei_ucs2le

ie UCS2-LE (UTF-16 Little Endian), which should match Windows internal "Unicode" encoding.

http://en.wikipedia.org/wiki/UTF-16/UCS-2

However you are clearly recommended to explicitly specify UCS2-LE or UCS2-BE unless the first bytes are a Byte Order Mark (BOM) value 0xfeff to indicate byte order scheme.

=> You are seeing the BOM as the first bytes of the output because that is what the "UNICODE" encoding name means, it means UCS2 with a header indicating the byte order scheme.