How to exec mmaped binary/code right from the allocated memory [closed]

Answer 1

It's next to impossible to exec into an arbitrary memory image. ELF files must be mmaped in a particular way in the process address space, almost never as a contiguous image of the underlying file.

The closest workable approach I can think of is saving the image into a (possibly nameless) file and exec'ing that file via execveat(fd, "", ...). Check execveat(2) around where they talk about empty pathname, and open(2) on O_TMPFILE.

A named temp file and a regular execve(2) would work as well, as long as the gap between close/munmap and execve when the file may be changed is not an issue.

Another idea would be to do ELF loading completely in userspace, setting up the proper address space for the new process, calling LD and such. Definitely not easy. Does not require any execve(2) call, just fork and work from there. Because execve(2) won't be used, certain things like O_CLOEXEC handling and un-mmaping extra memory will require non-trivial workarounds.

Answer 2

While your question remains confusing even after the edit, I think you've said enough to identify what you're doing wrong.

any binary(c hello world)

You cannot execute a mmaped binary file in the same way that the OS executes it. You're running on Linux so your hello world is likely a binary in the ELF format. Linux knows how to read and execute ELF binaries, your program doesn't. The ELF starts with a header and so your memory-mapped data also starts with the header. Which is not executable code. The header starts with 0x7F, which corresponds to the jg instruction on x86, so attempting to 'execute' an ELF header as executable code would possibly (depending on the status of your flags) immediately jump to a garbage address built up of the next bytes of the header.

If you want to run code like this, you'll need to parse the ELF file, find its entry point (or, likely, the .init section first) and start executing code from there. You should certainly feel free to do this as an exercise, for learning purposes, but that is not the normal way of executing another program from your own. A typical approach would be to use the fork and exec calls instead. That lets Linux handle the technicalities of launching the program.