How would i make this a decryption instead of an encryption?

Question

Wanna know how to get this from being a encryption code and using the same code to create a decryption.

I know it means that I have to reverse some of the instruction an

Answer 1

If this is a homework assignment, then it's actually quite a tricky one. (Though I may have overlooked a simple approach.)

Let's focus on the effect this code has on memory. For simplicity I will assume you are not interested in the effect this code has on registers, though I cannot be sure within seeing the code calling this function.

encryptL:
    push  eax
    ...                ; ignoring this code because its effect on eax is undone by the following pop
    pop   eax
    ...                ; code not affecting eax
    movzx ecx, [eax]
    ...                ; code not affecting eax
    mov   [eax], cl
    ret

So in goes an address (eax). The byte at this address is put into cl, and after some transformations, cl is put back at the same address. We need to figure out what those transformations are, and invert those.

    ror   cl, 1
    xor   cl, 0x96
    push  ecx
    ...                ; ignoring this code because its effect on ecx is undone by the following pop
    pop   ecx
    xor   ecx, edx

So a rotate and two xors are applied to cl. Xor is an involution, so we do not have to change those two instructions. The inverse of rol is of course ror, though that is not entirely true. The effect on the carry flag cannot be inverted, but in this case that effect is ignored, so we are good.

But what about all the code we ignored? On close inspection, you'll notice it is only there to calculate the value of edx that is used in xor ecx,edx. Unfortunately, the original (unencrypted) value of [eax] is involved in that calculation. When decrypting, the original value becomes the final value, i.e. the result of xor ecx,edx. We seem to have a chicken-and-egg problem. How to perform the calculation when it depends on the result of that same calculation?

It should be clear that the effect of [eax] on the calculation of edx, is either 1 out of only 8 possible cases, due to this statement:

    and   cl, 0x7

That means we can use a loop to try all different possibilities, until we found one where the result of the calculation matches the value we found in [eax]. That loop will be very similar to this loop found in the encryptor:

X:
    add   dl, 2
    sub   cl, 1
    jg    X

However, the exit condition of the loop will be different.

    mov   bl,0        ; using bl as a loop counter (similar to cl in original loop)
X:
    inc   bl
    add   dl,2        ; same as in the original loop
    mov   cl,[eax]    ; fetch encrypted byte from memory
    xor   ecx,edx     ; try transforming cl with the current value of dl
    xor   ebx,ecx     ; compare cl with the loop counter (bl)
    and   bl,7        ; only compare the 3 least significant bits
    jg    X           ; if not zero, try again with the next possible value of dl

Since xor is commutative and associative, we can rewrite this code. Instead of incrementing bl until it matches cl, we will decrement cl until its (partially) transformed value is zero (modulo 8).

    mov   cl,[eax]    ; fetch encrypted byte from memory
X:
    add   dl,2        ; same as in the original loop
    sub   cl,1        ; same as in the original loop
    push  ecx
    xor   ecx,edx     ; transform cl with the current value of dl
    and   cl,7        ; only consider the 3 least significant bits
    pop   ecx         ; restore cl (without clobbering flags)
    jg    X           ; if equal, then we found the right starting point

As for the rest of the code, that can stay as it is. It is mainly the initial part of the calculation of edx, before [eax] becomes involved.

Here is the complete solution:

encryptL:
    push  eax
    xchg  eax, ecx
    neg   al
    inc   eax
    rol   al, 1
    rol   al, 1
    mov   ebx, eax
    pop   eax
    push  ebx
    pop   edx

    movzx ecx, [eax]
    push  ecx
X:
    add   dl,2
    sub   cl,1
    push  ecx
    xor   ecx,edx
    and   cl,7
    pop   ecx
    jg    X

    pop   ecx
    xor   ecx,edx
    xor   cl,0x96
    rol   cl,1
    mov   [eax],cl
    ret

Disclaimer: I did not test it, so it's quite likely it will not work. I will leave the debugging up to you.