dlopen from memory?

Question

I\'m looking for a way to load generated object code directly from memory.

I understand that if I write it to a file, I can call dlopen to dynamically load its symbo

Answer 1

You don't need to load the code generated in memory, since it is already in memory!

However, you can -in a non portable way- generate machine code in memory (provided it is in a memory segment mmap-ed with PROT_EXEC flag).

^{(in that case, no "linking" or relocation step is required, since you generate machine code with definitive absolute or relative addresses, in particular to call external functions)}

Some libraries exist which do that: On GNU/Linux under x86 or x86-64, I know of GNU Lightning (which generates quickly machine code which runs slowly), DotGNU LibJIT (which generates medium quality code), and LLVM & GCCJIT (which is able to generate quite optimized code in memory, but takes time to emit it). And LuaJit has some similar facility too. Since 2015 GCC 5 has a gccjit library.

And of course, you can still generate C code in a file, fork a compiler to compile it into a shared object, and dlopen that shared object file. I'm doing that in GCC MELT , a domain specific language to extend GCC. It does work quite well in practice.

addenda

If performance of writing the generated C file is a concern (it should not be, since compiling a C file is much slower than writing it) consider using some tmpfs file system for that (perhaps in /tmp/ which is often a tmpfs filesystem on Linux)

Answer 2

There is no standard way to do it other than writing out the file and then loading it again with dlopen().

You may find some alternative method on your current specific platform. It will up to you to decide whether that is better than using the 'standard and (relatively) portable' approach.

Since generating the object code in the first place is rather platform specific, additional platform-specific techniques may not matter to you. But it is a judgement call - and in any case depends on there being a non-standard technique, which is relatively improbable.

Answer 3

I needed a solution to this because I have a scriptable system that has no filesystem (using blobs from a database) and needs to load binary plugins to support some scripts. This is the solution I came up with which works on FreeBSD but may not be portable.

void *dlblob(const void *blob, size_t len) {
    /* Create shared-memory file descriptor */
    int fd = shm_open(SHM_ANON, O_RDWR, 0);
    ftruncate(fd, len);
    /* MemMap file descriptor, and load data */
    void *mem = mmap(NULL, len, PROT_WRITE, MAP_SHARED, fd, 0);
    memcpy(mem, blob, len);
    munmap(mem, len);
    /* Open Dynamic Library from SHM file descriptor */
    void *so = fdlopen(fd,RTLD_LAZY);
    close(fd);
    return so;
}

Obviously the code lacks any kind of error checking etc, but this is the core functionality.

ETA: My initial assumption that fdlopen is POSIX was wrong, this appears to be a FreeBSD-ism.

Answer 4

I don't see why you'd be considering dlopen, since that will require a lot more nonportable code to generate the right object format on disk (e.g. ELF) for loading. If you already know how to generate machine code for your architecture, just mmap memory with PROT_READ|PROT_WRITE|PROT_EXEC and put your code there, then assign the address to a function pointer and call it. Very simple.