Char C question about encoding signed/unsigned

Question

I read that C not define if a char is signed or unsigned, and in GCC page this says that it can be signed on x86 and unsigned in PowerPPC and ARM.

Okey, I'm writing a program with GLIB that define char as gchar (not more than it, only a way for standardization).

My question is, what about UTF-8? It use more than an block of memory?

Say that I have a variable

unsigned char *string = "My string with UTF8 enconding ~> çã";

See, if I declare my variable as

unsigned

I will have only 127 values (so my program will to store more blocks of mem) or the UTF-8 change to negative too?

Sorry if I can't explain it correctly, but I think that i is a bit complex.

NOTE: Thanks for all answer

I don't understand how it is interpreted normally.

I think that like ascii, if I have a signed and unsigned char on my program, the strings have diferently values, and it leads to confuse, imagine it in utf8 so.

Answer 1

I've had a couple requests to explain a comment I made.

The fact that a char type can default to either a signed or unsigned type can be significant when you're comparing characters and expect a certain ordering. In particular, UTF8 uses the high bit (assuming that char is an 8-bit type, which is true in the vast majority of platforms) to indicate that a character code point requires more than one byte to be represented.

A quick and dirty example of the problem:

#include <stdio.h>
int main( void)
{
    signed char flag = 0xf0;
    unsigned char uflag = 0xf0;

    if (flag < (signed char) 'z') {
        printf( "flag is smaller than 'z'\n");
    }
    else {
        printf( "flag is larger than 'z'\n");
    }    


    if (uflag < (unsigned char) 'z') {
        printf( "uflag is smaller than 'z'\n");
    }
    else {
        printf( "uflag is larger than 'z'\n");
    }
    return 0;
}

On most projects that I work, the unadorned char type is typically avoided in favor us using a typedef that explicitly specifies an unsigned char . Something like the uint8_t from stdint.h or

typedef unsigned char u8;

Generally dealing with an unsigned char type seems to work well and have few problems - the one area that I have seen occasional problems is when using something of that type to control a loop:

while (uchar_var-- >= 0) {
    // infinite loop...
}

Answer 2

Using unsigned char has its pros and cons. The biggest benefits are that you don't get sign extension or other funny features such as signed overflow that would produce unexpected results from calculations. Unsigned char is also compatible with <cctype> macros/functions such as isalpha(ch) (all these require values in unsigned char range). On the other hand, all I/O functions require char*, requiring you to cast whenever you do I/O.

As for UTF-8, storing it in signed or unsigned arrays is fine but you have to be careful with those string literals as there is little guarantee about them being valid UTF-8. C++0x adds UTF-8 string literals to avoid possible issues and I would expect the next C standard to adopt those as well.

In general you should be fine, though, as long as you make sure that your source code files are always UTF-8 encoded.

Answer 3

Two things:

1. Whether a char type is signed or unsigned won't affect your ability to translate UTF8-encoded-strings to and from whatever display string type you're using (WCHAR or whatnot). Don't worry about it, in other words: the UTF8 bytes are just bytes, and whatever you're using as an encoder/decoder will do the right thing.

2. Some of your confusion may be that you're trying to do this:

    unsigned char *string = "This is a UTF8 string"; 

   Don't do this-- you're mixing different concepts. A UTF-8 encoded string is just a sequence of bytes. C string literals (as above) were not really designed to represent this; they're designed to represent "ASCII-encoded" strings. Although for some cases (like mine here) they end up being the same thing, in your example in the question, they may not. And certainly in other cases they won't be. Load your Unicode strings from an external resource. In general I'd be wary of embedding non-ASCII characters in a .c source file; even if the compiler knows what to do with them, other software in your toolchain may not.

Answer 4

signed / unsigned affect only arithmetic operations. if char is unsigned then higher values will be positive. in case of signed they will be negative. But range is same still.

Answer 5

Not really, unsigned / signed does not specify how many values a variable can hold. It specifies how they are interpreted .

So, an unsigned char has the same amount of values as a signed char , except that the one has negative numbers and the other doesn't. It is still 8 bits (if we assume that a char holds 8 bits, I'm not sure it does everywhere).

Answer 6

It makes no differences when using a char* as a string. The only time signed/unsigned would make a difference is if you would be interpreting it as a number, like for arithmetic or if you were to print it as an integer.

Answer 7

UTF-8 characters cannot be assumed to store in one byte. UTF-8 characters can be 1-4 bytes wide. So, a char , wchar_t , signed or unsigned would not be sufficient for assuming one unit can always store one UTF-8 character.

Most platforms (such as PHP, .NET, etc.) have you build strings normally (such as char[] in C) and you use a library to convert between encodings and parse characters out of the string.

Answer 8

As to you'r question:

think if I have a singed or unsigned ARRAY of chars can be it make my program run wrong? – drigoSkalWalker

Yes. Mine did. Heres a simple runnable excerpt from my app that totally comes out wrong if using ordinary signed chars. Try running it after changing all chars to unsigned in parameters. Like this:

int is_valid( unsigned char c);

it should then work properly.

#include <stdio.h>

int is_valid(char c);

int main() {

    char ch = 0xFE;
    int ans = is_valid(ch);
    printf("%d", ans);

}

int is_valid(char c) {
    if((c == 0xFF) || (c == 0xFE)) {
    printf("NOT valid\n");
        return 0;
    }
    else {
        printf("valid\n")
        return 1;
    }
}  

What it does is validate if the char is a valid byte within utf-8. 0xFF and 0xFE are NOT valid bytes in utf-8. imagine the problem if the function validates it as a valid byte?

what happens is this:

0xFE
= 
11111110 
= 
254

If you save this in a ordinary char (that is signed) the leftmost bit, most significant bit, makes it negative. But what negative number is it?

It does this by flipping the bits and adding one bit.

11111110
00000001
00000001 + 00000001 =
00000010 = 2

and remember it made it negative, so it becomes -2

so (-2 == 0xFE) in the function ofcourse isnt true. same goes for (-2 == 0xFF).

So a function that checks for invalid bytes ends up validating unvalid bytes as if they are ok :-o.

Two other reasons I can think of to stick to unsigned when dealing with utf-8 is:

1. If you might need some bitshifting to the right, there can be trouble because then you might end up adding 1's from the left if using signed chars.

2. utf-8 and unicode only uses positive numbers so... why dont you as well? keeping it simple :)