Is it possible to “punch holes” through mmap'ed anonymous memory?

Question

Consider a program which uses a large number of roughly page-sized memory regions (say 64 kB or so), each of which is rather short-lived. (In my particular case, these are alter

Answer 1

I don't see why doing lots of calls to mmap/munmap should be that bad. The lookup performance in the kernel for mappings should be O(log n).

Your only options as it seems to be implemented in Linux right now is to punch holes in the mappings to do what you want is mprotect(PROT_NONE) and that is still fragmenting the mappings in the kernel so it's mostly equivalent to mmap/munmap except that something else won't be able to steal that VM range from you. You'd probably want madvise(MADV_REMOVE) work or as it's called in BSD - madvise(MADV_FREE). That is explicitly designed to do exactly what you want - the cheapest way to reclaim pages without fragmenting the mappings. But at least according to the man page on my two flavors of Linux it's not fully implemented for all kinds of mappings.

Disclaimer: I'm mostly familiar with the internals of BSD VM systems, but this should be quite similar on Linux.

As in the discussion in comments below, surprisingly enough MADV_DONTNEED seems to do the trick:

#include 
#include 
#include 
#include 

#include 
#include 

#include 

int
main(int argc, char **argv)
{
        int ps = getpagesize();
        struct rusage ru = {0};
        char *map;
        int n = 15;
        int i;

        if ((map = mmap(NULL, ps * n, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)) == MAP_FAILED)
                err(1, "mmap");

        for (i = 0; i < n; i++) {
                map[ps * i] = i + 10;
        }

        printf("unnecessary printf to fault stuff in: %d %ld\n", map[0], ru.ru_minflt);

        /* Unnecessary call to madvise to fault in that part of libc. */
        if (madvise(&map[ps], ps, MADV_NORMAL) == -1)
                err(1, "madvise");

        if (getrusage(RUSAGE_SELF, &ru) == -1)
                err(1, "getrusage");
        printf("after MADV_NORMAL, before touching pages: %d %ld\n", map[0], ru.ru_minflt);

        for (i = 0; i < n; i++) {
                map[ps * i] = i + 10;
        }

        if (getrusage(RUSAGE_SELF, &ru) == -1)
                err(1, "getrusage");
        printf("after MADV_NORMAL, after touching pages: %d %ld\n", map[0], ru.ru_minflt);

        if (madvise(map, ps * n, MADV_DONTNEED) == -1)
                err(1, "madvise");

        if (getrusage(RUSAGE_SELF, &ru) == -1)
                err(1, "getrusage");
        printf("after MADV_DONTNEED, before touching pages: %d %ld\n", map[0], ru.ru_minflt);

        for (i = 0; i < n; i++) {
                map[ps * i] = i + 10;
        }

        if (getrusage(RUSAGE_SELF, &ru) == -1)
                err(1, "getrusage");
        printf("after MADV_DONTNEED, after touching pages: %d %ld\n", map[0], ru.ru_minflt);

        return 0;
}

I'm measuring ru_minflt as a proxy to see how many pages we needed to allocate (this isn't exactly true, but the next sentence makes it more likely). We can see that we get new pages in the third printf because the contents of map[0] are 0.