How to write self-modifying code in x86 assembly

Question

I\'m looking at writing a JIT compiler for a hobby virtual machine I\'ve been working on recently. I know a bit of assembly, (I\'m mainly a C programmer. I can read most ass

Answer 1

You can also look at projects like GNU lightning. You give it code for a simplified RISC-type machine, and it generates correct machine dynamically.

A very real problem you should think about is interfacing with foreign libraries. You will probably need to support at least some system-level calls/operations for your VM to be useful. Kitsune's advice is a good start to get you thinking about system-level calls. You would probably use mprotect to ensure that the memory you have modified becomes legally executable. (@KitsuneYMG)

Some FFI allowing calls to dynamic libraries written in C should be sufficient to hide a lot of the OS specific details. All these issues can impact your design quite a bit, so it is best to start thinking about them early.