Trying to reverse engineer a function

Question

I\'m trying to understand assembly in x86 more. I have a mystery function here that I know returns an int and takes an int argument. So it looks l

Answer 1

The LEA performs an address computation, but instead of dereferencing the address, it stores the computed address into the destination register. In AT&T syntax, lea C(b,c,d), reg means reg = C + b + c*d where C is a constant, and b,c are registers and d is a scalar from {1,2,4,8}. Hence you can see why LEA is popular for simple math operations: it does quite a bit in a single instruction. (*includes correction from prl's comment below)

There are some strange features of this assembly code: the repz prefix is only strictly defined when applied to certain instructions, and retq is not one of them (though the general behavior of the processor is to ignore it). See Michael Petch's comment below with a link for more info. The use of lea (,rdi,8), edi followed by sub eax, edi to compute arg1 * 7 also seemed strange, but makes sense once prl noted the scalar d had to be a constant power of 2. In any case, here's how I read the snippet:

mov  %edi, %eax          ; eax = arg1
lea  0x0(,%rdi, 8), %edi ; edi = arg1 * 8
sub  %eax, %edi          ; edi = (arg1 * 8) - arg1 = arg1 * 7
add  $0x4, %edi          ; edi = (arg1 * 7) + 4
callq < mystery _util >  ; call mystery_util(arg1 * 7 + 4)
repz retq                ; repz prefix on return is de facto nop.


< mystery _util >
mov  %edi, %eax          ; eax = arg1
shr  %eax                ; eax = arg1 >> 1
and  $0x1, %edi          ; edi = 1 iff arg1 was odd, else 0
and  %edi, %eax          ; eax = 1 iff smallest 2 bits of arg1 were both 1.
retq

Note the +4 on the 4th line is entirely spurious. It cannot affect the outcome of mystery_util.

So, overall this ASM snippet computes the boolean (arg1 * 7) % 4 == 3.