How to do Binary instrumentation of syscall brk ? (x86-64 Linux) (maybe valgrind?)

Question

I\'d like to instrument syscall brk (and other calls but this in first order, it\'s most important to me) in given binary (preferably on actual syscall/sysenter lev

Answer 1

LD_PRELOAD will trap C calls to brk(), but it won't trap the actual system call (int/syscall instruction). There's no portable way to trap those, but on Linux, ptrace will do it. Memory can also be allocated to a program by mmap(), so you'll need to intercept that call too.

Of course, what it seems you're really looking for is rlimit().

Answer 2

Yeah, I don't think you want valgrind for this.

You can use LD_PRELOAD or linker tricks to capture brk(2): see these other discussions:

Function interposition in Linux without dlsym

Overriding 'malloc' using the LD_PRELOAD mechanism

Code might look like this:

#include <unistd.h>
#include <dlfcn.h>

/* prototype int brk(void *addr); */

static int (*real_brk)(void *addr) = NULL;

int brk(void * addr) {

    real_brk = dlsym(RTLD_NEXT, "brk");
    if (real_brk == NULL) {
            fprintf(stderr, "error mapping brk: %s\n", dlerror());
            return -1;
    }
    printf("calling brk(2) for %p\n", addr);
    return (real_brk (addr));
}`   

and then LD_PRELOAD that to intercept brk(2)