I am reading some code that implements a simple parser. A function named scan
breaks up a line into tokens. scan
has a static variable bp
that is assigned the line to be tokenized. Following the assignment, the whitespace is skipped over. See below. What I don't understand is why the code does a bitwise-and of the character that bp
points to with 0xff
, ie, what is the purpose of * bp & 0xff
? How is this:
while (isspace(* bp & 0xff))
++ bp;
different from this:
while (isspace(* bp))
++ bp;
Here is the scan
function:
static enum tokens scan (const char * buf)
/* return token = next input symbol */
{ static const char * bp;
while (isspace(* bp & 0xff))
++ bp;
..
}
From the C Standard (7.4 Character handling <ctype.h>)
1 The header <ctype.h> declares several functions useful for classifying and mapping characters.198) In all cases the argument is an int, the value of which shall be representable as an unsigned char or shall equal the value of the macro EOF. If the argument has any other value, the behavior is undefined.
In this call
isspace(* bp)
the argument expression *bp
having the type char
is converted to the type int
due to the integer promotions.
If the type char
behaves as the type signed char
and the value of the expression *bp
is negative then the value of the promoted expression of the type int
is also will be negative and can not be representable as a value of the type unsigned char
.
This results in undefined behavior.
In this call
isspace(* bp & 0xff)
due to the bitwise operator & the result value of the expression * bp & 0xff
of the type int
can be represented as a value of the type unsigned char
.
So it is a trick used instead of writing a more clear code like
isspace( ( unsigned char )*bp )
The function isspace
is usually implemented such a way that it uses its argument of the type int
as an index in a table with 256 values (from 0 to 255). If the argument of the type int
has a value that is greater than the maximum value 255 or a negative value (and is not equal to the value of the macro EOF) then the behavior of the function is undefined.
From cppreference isspace() : The behavior is undefined if the value of ch is not representable as unsigned char and is not equal to EOF
.
When *bp
is negative, for example it's -42
, then it is not representable as unsigned char
, because it's negative and unsigned char
, well, must be positive or zero.
On twos-complement systems values aresign extended to bigger "width", so then they will get left-most bits set. Then when you take 0xff
of the wider type, the left-most bits are cleared, and you end up with a positive value, lower or equal to 0xff
, I mean representable as unsigned char
.
Note that arguments to &
undergo implicit promotions , so the result of *bp
is converted to int
before even calling isspace
. Let's assume that *bp = -42
for example and assume a sane platform with 8-bit char that is signed and that int
has 32-bits, then:
*bp & 0xff # expand *bp = -42
(char)-42 & 0xff # apply promotion
(int)-42 & 0xff # lets convert to hex assuming twos-complement
(int)0xffffffd6 & 0xff # do & operation
(int)0xd6 # lets convert to decimal
214 # representable as unsigned char, all fine
Without the & 0xff
the negative value would result in undefined behavior.
I would recommend to prefer isspace((unsigned char)*bp)
.
Basically the simplest isspace
implementation looks like just :
static const char bigarray[257] = { 0,0,0,0,0,...1,0,1,0,... };
// note: EOF is -1
#define isspace(x) (bigarray[(x) + 1])
and in such case you can't pass for example -42
, cause bigarray[-41]
is just invalid.
Your question:
How is this:
while (isspace(* bp & 0xff))
++ bp;
different from this:
while (isspace(* bp))
++ bp;
The difference is, in the first example you are always passing the lowermost byte at bp
to isspace
, due to the result of a bitwise AND with a full bitmask ( 0b11111111
or 0xff
). It's possible that the argument to isspace
contains a type that is larger than 1 byte. For example, isspace
is defined as isspace(int c)
, so as you can see the argument here is an int
, which may be multiple bytes depending on your system.
In short, it's a sanity check to ensure that isspace
is only comparing a single byte from your bp
variable.
while (isspace(* bp & 0xff))
++ bp;
&&
while (isspace(* bp))
++ bp;
Strictly speaking, both are incorrect if bp
does not reference unsigned char
.
In this case it should be:
while (isspace((unsigned char)(*bp & 0xff)))
++ bp;
or better
while (isspace(*bp == EOF ? EOF : (unsigned char)(*bp & 0xff)))
++ bp;
isspace is undefined if parameter is not EOF
or it does not have the value of unsigned char
if *bp
references char
it has to be:
while (isspace((unsigned char)(*bp)))
++bp;
In c char can be signed or unsigned https://en.wikipedia.org/wiki/C_data_types
When passed to isspace
, bp
will be promoted to an integer. If it is signed and the high bit is set then it will be sign extended to become a negative integer. This may mean it is not an unsigned char or EOF as required by the isspace
function https://linux.die.net/man/3/isspaceNo
See http://cpp.sh/9mp2i for how it changes the bitwise and changes value of that isspace sees
If we assume bits of char type are always 8,
then the code bitwise-and operator with 0xff here will confuse us.
But what about that if char type is not always 8-bits?
Then 0xff may have another meaning, right?
Actually, the char type is not always 8-bits and we can see the detail in C99 standard. The char type in standard is not defined as 8 bits.
The following is how C99 standard describe the size of char type.
6.5.3.4 The sizeof operator When applied to an operand that has type char , unsigned char , or signed char , (or a qualified version thereof) the result is 1 . When applied to an operand that has array type, the result is the total number of bytes in the array.) When applied to an operand that has structure or union type, the result is the total number of bytes in such an object, including internal and trailing padding.
6.2.5 Types An object declared as type char is large enough to store any member of the basic execution character set . If a member of the basic execution character set is stored in a char object, its value is guaranteed to be positive. If any other character is stored in a char object, the resulting value is implementation-defined but shall be within the range of values that can be represented in that type.
For example, TMS320C28x DSP from Texas Instruments has a char with 16 bits.
For the compiler specifies here , CHAR_BIT as 16 on page 99.
This appears to be a modern processor (currently being sold), compilers supporting C99 and C++03.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.