How to do Binary instrumentation of syscall brk ? (x86-64 Linux) (maybe valgrind?)

Question

I\'d like to instrument syscall brk (and other calls but this in first order, it\'s most important to me) in given binary (preferably on actual syscall/sysenter lev

Answer 1

LD_PRELOAD will trap C calls to brk(), but it won't trap the actual system call (int/syscall instruction). There's no portable way to trap those, but on Linux, ptrace will do it. Memory can also be allocated to a program by mmap(), so you'll need to intercept that call too.

Of course, what it seems you're really looking for is rlimit().