This C function should always return false, but it doesn’t

Question

I stumbled over an interesting question in a forum a long time ago and I want to know the answer.

Consider the following C function:

f1.c

#incl         


        

Answer 1

I think it's interesting to see where the size-mismatch mentioned in Lundin's excellent answer actually happens.

If you compile with --save-temps, you will get assembly files that you can look at. Here's the part where f1() does the == 0 comparison and returns its value:

cmpl    $0, -4(%rbp)
sete    %al

The returning part is sete %al. In C's x86 calling conventions, return values 4 bytes or smaller (which includes int and bool) are returned via register %eax. %al is the lowest byte of %eax. So, the upper 3 bytes of %eax are left in an uncontrolled state.

Now in main():

call    f1
testl   %eax, %eax
je  .L2

This checks whether the whole of %eax is zero, because it thinks it's testing an int.

Adding an explicit function declaration changes main() to:

call    f1
testb   %al, %al
je  .L2

which is what we want.