How cmp assembly instruction sets flags (X86_64 GNU Linux)

Question

Here is a simple C program:

void main()
{
       unsigned char number1 = 4;
       unsigned char number2 = 5;

       if (number1 < number2)
       {
             


        

Answer 1

I think i understand it now. This is how i think it goes (borrow flag is set)

4 - 5

1st operand = 4 = 0000 0100
2nd operand = 5 = 0000 0101

So we have to perform

      1st operand
    - 2nd operand
    --------------


      7654 3210 <-- Bit number
      0000 0100
    - 0000 0101
    ------------

Lets start.

Bit 0 of 1st operand = 0
Bit 0 of 2nd operand = 1

so

  0
- 1 
 ===
  ?

to do this,

let's borrow a 1 from left side of bit 0 of 1st operand.

so we see bit 2 of 1st operand is 1.

when bit 2 is = 1, it means 4.

we know that we can write 4 as 2 + 2. So we can write 4 as two 2s.

      7654 3210 <-- Bit number
             1
             1         
      0000 0000
    - 0000 0101
    ------------

So in above step, we have written bit 4 of 1st operand as two 2s (two 1 on top of bit 2 of 1st operand.)

Now again as we know, a 2 can be written as two 1s. So we borrow one 1 from bit 1 of 1st operand and write two 1s on bit 0 of 1st operand.

      7654 3210 <-- Bit number
              1
             11         
      0000 0000
    - 0000 0101
    ------------

Now we are ready to perform subtraction on bit 0 and bit 1.

      7654 3210 <-- Bit number
              1
             11         
      0000 0000
    - 0000 0101
    ------------
             11

So after solving bit 0 and bit 1, lets see bit 2.

We again see same problem.

Bit 2 of 1st operand = 0

Bit 2 of 2nd operand = 1

to do this, let's borrow a 1 from left side of bit 2 of 1st operand.

    8 7654 3210 <-- Bit number
              1
             11         
    1 0000 0000
    - 0000 0101
    ------------
             11

Now you see, bit 8 of 1st operand is 1. We have borrowed this 1.

At this stage, carry flag will be set. So CF=1.

Now, if bit 8 is 1, it means 256.

256 = 128 + 128

if bit 7 is 1, it means 128. We can rewrite as

    8 7654 3210 <-- Bit number
      1       1
      1      11         
      0000 0000
    - 0000 0101
    ------------
             11

As previously, we can re-write it as:

    8 7654 3210 <-- Bit number
       1      1
      11     11         
      0000 0000
    - 0000 0101
    ------------
             11

As previously, we can re-write it as:

    8 7654 3210 <-- Bit number
        1     1
      111    11         
      0000 0000
    - 0000 0101
    ------------
             11

As previously, we can re-write it as:

    8 7654 3210 <-- Bit number
         1    1
      1111   11         
      0000 0000
    - 0000 0101
    ------------
             11

As previously, we can re-write it as:

    8 7654 3210 <-- Bit number
           1  1
      1111 1 11         
      0000 0000
    - 0000 0101
    ------------
             11

As previously, we can re-write it as:

    8 7654 3210 <-- Bit number
            1 1
      1111 1111         
      0000 0000
    - 0000 0101
    ------------
             11

At last we can solve this.

Subtracting 2nd operand from all above it will give

    8 7654 3210 <-- Bit number
            1 1
      1111 1111         
      0000 0000
    - 0000 0101
    ------------
      1111 1111


So result = 1111 1111

Notice, sign bit in result = bit 7 = 1

so sign flag will be set. i.e SF=1

And therefore SF=1, CF=1 after 4 - 5

Answer 2

Operation of CMP
CMP performs a subtraction but does not store the result.
For this reason the effect on the flags is exactly the same between:

cmp eax,ecx
sub eax,ecx

As per the documentation:

Operation
temp ← SRC1 − SignExtend(SRC2);
ModifyStatusFlags; (* Modify status flags in the same manner as the SUB instruction*)
Flags Affected
The CF, OF, SF, ZF, AF, and PF flags are set according to the result.

Effects on the flags
So the following flags are affected like so:

Assume result = op1 - op2

CF - 1 if unsigned op2 > unsigned op1
OF - 1 if sign bit of OP1 != sign bit of result
SF - 1 if MSB (aka sign bit) of result = 1
ZF - 1 if Result = 0 (i.e. op1=op2)
AF - 1 if Carry in the low nibble of result
PF - 1 if Parity of Least significant byte is even

I suggest you read up on the OF and CF here: http://teaching.idallen.com/dat2343/10f/notes/040_overflow.txt

Order of the operands
I see that you like pain and are using the braindead variant of x86 assembly called ATT-syntax.
This being the case you need to take into account that

CMP %EAX, %ECX  =>  result for the flags = ECX - EAX
CMP OP2, OP1    =   flags = OP1 - OP2

Whereas Intel syntax is

CMP ECX, EAX    =>  result for the flags = ECX - EAX
CMP OP1, OP2    =>  flags = OP1 - OP2

You can instruct gdb to show you Intel syntax using: set disassembly-flavor intel