How to write self-modifying code in x86 assembly

Question

I\'m looking at writing a JIT compiler for a hobby virtual machine I\'ve been working on recently. I know a bit of assembly, (I\'m mainly a C programmer. I can read most ass

Answer 1

wow, this turned out to be a lot more painful than I expected. 100% of the pain was linux protecting the program from being overwritten and/or executing data.

Two solutions shown below. And a lot of googling was involved so the somewhat simple put some instruction bytes and execute them was mine, the mprotect and aligning on page size was culled from google searches, stuff I had to learn for this example.

The self modifying code is straight forward, if you take the program or at least just the two simple functions, compile and then disassemble you will get the opcodes for those instructions. or use nasm to compile blocks of assembler, etc. From this I determined the opcode to load an immediate into eax then return.

Ideally you simply put those bytes in some ram and execute that ram. To get linux to do that you have to change the protection, which means you have to send it a pointer that is aligned on a mmap page. So allocate more than you need, find the aligned address within that allocation that is on a page boundary and mprotect from that address and use that memory to put your opcodes and then execute.

the second example takes an existing function compiled into the program, again because of the protection mechanism you cannot simply point at it and change bytes, you have to unprotect it from writes. So you have to back up to the prior page boundary call mprotect with that address and enough bytes to cover the code to be modified. Then you can change the bytes/opcodes for that function in any way you want (so long as you don't spill over into any function you want to continue to use) and execute it. In this case you can see that fun() works, then I change it to simply return a value, call it again and now it has been modified.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>

unsigned char *testfun;

unsigned int fun ( unsigned int a )
{
    return(a+13);
}

unsigned int fun2 ( void )
{
    return(13);
}

int main ( void )
{
    unsigned int ra;
    unsigned int pagesize;
    unsigned char *ptr;
    unsigned int offset;

    pagesize=getpagesize();
    testfun=malloc(1023+pagesize+1);
    if(testfun==NULL) return(1);
    //need to align the address on a page boundary
    printf("%p\n",testfun);
    testfun = (unsigned char *)(((long)testfun + pagesize-1) & ~(pagesize-1));
    printf("%p\n",testfun);

    if(mprotect(testfun, 1024, PROT_READ|PROT_EXEC|PROT_WRITE))
    {
        printf("mprotect failed\n");
        return(1);
    }

    //400687: b8 0d 00 00 00          mov    $0xd,%eax
    //40068d: c3                      retq

    testfun[ 0]=0xb8;
    testfun[ 1]=0x0d;
    testfun[ 2]=0x00;
    testfun[ 3]=0x00;
    testfun[ 4]=0x00;
    testfun[ 5]=0xc3;

    ra=((unsigned int (*)())testfun)();
    printf("0x%02X\n",ra);


    testfun[ 0]=0xb8;
    testfun[ 1]=0x20;
    testfun[ 2]=0x00;
    testfun[ 3]=0x00;
    testfun[ 4]=0x00;
    testfun[ 5]=0xc3;

    ra=((unsigned int (*)())testfun)();
    printf("0x%02X\n",ra);


    printf("%p\n",fun);
    offset=(unsigned int)(((long)fun)&(pagesize-1));
    ptr=(unsigned char *)((long)fun&(~(pagesize-1)));


    printf("%p 0x%X\n",ptr,offset);

    if(mprotect(ptr, pagesize, PROT_READ|PROT_EXEC|PROT_WRITE))
    {
        printf("mprotect failed\n");
        return(1);
    }

    //for(ra=0;ra&lt;20;ra++) printf("0x%02X,",ptr[offset+ra]); printf("\n");

    ra=4;
    ra=fun(ra);
    printf("0x%02X\n",ra);

    ptr[offset+0]=0xb8;
    ptr[offset+1]=0x22;
    ptr[offset+2]=0x00;
    ptr[offset+3]=0x00;
    ptr[offset+4]=0x00;
    ptr[offset+5]=0xc3;

    ra=4;
    ra=fun(ra);
    printf("0x%02X\n",ra);

    return(0);
}

Answer 2

This is written in AT&T assembly. As you can see from the execution of the program, output has changed because of self-modifying code.

Compilation: gcc -m32 modify.s modify.c

the -m32 option is used because the example works on 32 bit machines

Aessembly:

.globl f4
.data     

f4:
    pushl %ebp       #standard function start
    movl %esp,%ebp

f:
    movl $1,%eax # moving one to %eax
    movl $0,f+1  # overwriting operand in mov instuction over
                 # the new immediate value is now 0. f+1 is the place
                 # in the program for the first operand.

    popl %ebp    # standard end
    ret

C test-program:

 #include <stdio.h>

 // assembly function f4
 extern int f4();
 int main(void) {
 int i;
 for(i=0;i<6;++i) {
 printf("%d\n",f4());
 }
 return 0;
 }

Output:

1
0
0
0
0
0

Answer 3

Since you're writing a JIT compiler, you probably don't want self-modifying code, you want to generate executable code at runtime. These are two different things. Self-modifying code is code that is modified after it has already started running. Self-modifying code has a large performance penalty on modern processors, and therefore would be undesirable for a JIT compiler.

Generating executable code at runtime should be a simple matter of mmap()ing some memory with PROT_EXEC and PROT_WRITE permissions. You could also call mprotect() on some memory you allocated yourself, as dwelch did above.

Answer 4

I'm working on a self-modifying game to teach x86 assembly, and had to solve this exact problem. I used the following three libraries:

AsmJit + AsmTk for assembling: https://github.com/asmjit/asmjit + https://github.com/asmjit/asmtk UDIS86 for disassembling: https://github.com/vmt/udis86

Instructions are read with Udis86, the user can edit them as a string, and then AsmJit/AsmTk is used to assemble the new bytes. These can be written back to memory, and as other users have pointed out, the write-back requires using VirtualProtect on Windows or mprotect on Unix to fix the memory page permissions.

The code samples are a just a little long for StackOverflow, so I'll refer you to an article I wrote with code samples:

https://medium.com/squallygame/how-we-wrote-a-self-hacking-game-in-c-d8b9f97bfa99

A functioning repo is here (very light-weight):

https://github.com/Squalr/SelfHackingApp

Answer 5

A little bit simpler example based on the example above. Thanks to dwelch helped a lot.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/mman.h>

char buffer [0x2000];
void* bufferp;

char* hola_mundo = "Hola mundo!";
void (*_printf)(const char*,...);

void hola()
{ 
    _printf(hola_mundo);
}

int main ( void )
{
    //Compute the start of the page
    bufferp = (void*)( ((unsigned long)buffer+0x1000) & 0xfffff000 );
    if(mprotect(bufferp, 1024, PROT_READ|PROT_EXEC|PROT_WRITE))
    {
        printf("mprotect failed\n");
        return(1);
    }
    //The printf function has to be called by an exact address
    _printf = printf;

    //Copy the function hola into buffer
    memcpy(bufferp,(void*)hola,60 //Arbitrary size);


    ((void (*)())bufferp)();  

    return(0);
}

Answer 6

I've never written self-modifying code, although I have a basic understanding about how it works. Basically you write on memory the instructions you want to execute then jump there. The processor interpret those bytes you've written an instructions and (tries) to execute them. For example, viruses and anti-copy programs may use this technique.
Regarding the system calls, you were right, arguments are passed via registers. For a reference of linux system calls and their argument just check here.