printf和gcc -O选项更改返回值

Question

i'm trying to understand how gcc -O option and printf can change return value.

in the code below isTmax is my function that return 1 if the input is the largest int value.

int isTmax(int x);
int main(void)
{
    printf("%d\n", isTmax(0x7fffffff));
}

int isTmax(int x)
{
    int temp = x + x + 2;
    int result = !temp & (!!(~x));
    return result;
}

it worked perfectly when i compiled it without gcc option.

but with -O option, i got 0 for all integer values.

so i added printf to check values and then compiled it with -O option.

int isTmax(int x)
{
    int temp = x + x + 2;
    int result = !temp & (!!(~x));
    printf("x : %x, temp : %x, !temp : %x\n", x, temp , !temp); 
    return result;
}

And suddenly it's working again.

i want to know why the return value varies.

Answer 1

As it was pointed out in the comments, this is because of undefined behaviour.

Modern compilers can (ab)use undefined behaviour to simplify, shorten, and speed up the compiled code without violating any rules. I suggest you read that excellent article about the dangers of UB.

It's hard to know what exactly is going on because optimizer are becoming exceedingly complex, but here's a take on what the compiler might be doing with your function.

(1) int isTmax(int x)
(2) {
(3)     int temp = x + x + 2;
(4)     int result = !temp & (!!(~x));
(5)     return result;
(6) }

On line 3, you're adding together two signed integers. If your program only call this function once with 0x7fffffff , the compiler already knows that the code only produces an integer overflow. There's a twist with that code ; since you add 2 to an overflowing operation, the compiler can assume that the value will be positive and greater than two for the sake of what comes next.

On line 4, !temp translate to the boolean constant false since temp is assumed to be a positive value. Next, the and ; since the left value is false (or 0), result always end up being 0. If the result is always zero, the optimizer might as well remove all the variables and operations.

So essentially, after optimization, your function looks like :

int isTmax(int x)
{
    return 0;
}

By adding a printf inside the function, you're forcing the optimizer into de-optimizing, so you end up with the same code you get when you don't optimize your program.