Can address space be recycled for multiple calls to MapViewOfFileEx without chance of failure?

Question

Consider a complex, memory hungry, multi threaded application running within a 32bit address space on windows XP.

Certain operations require n large buffers of fixed

Answer 1

Imagine if I came to you with a piece of code like this:

void *foo;

foo = malloc(n);
if (foo)
   free(foo);
foo = malloc(n);

Then I came to you and said, help! foo does not have the same address on the second allocation!

I'd be crazy, right?

It seems to me like you've already demonstrated clear knowledge of why this doesn't work. There's a reason that the documention for any API that takes an explicit address to map into lets you know that the address is just a suggestion, and it can't be guaranteed. This also goes for mmap() on POSIX.

I would suggest you write the program in such a way that a change in address doesn't matter. That is, don't store too many pointers to quantities inside the buffer, or if you do, patch them up after reallocation. Similar to the way you'd treat a buffer that you were going to pass into realloc().

Even the documentation for MapViewOfFileEx() explicitly suggests this:

While it is possible to specify an address that is safe now (not used by the operating system), there is no guarantee that the address will remain safe over time. Therefore, it is better to let the operating system choose the address. In this case, you would not store pointers in the memory mapped file, you would store offsets from the base of the file mapping so that the mapping can be used at any address.

Update from your comments

In that case, I suppose you could:

• Not map into contiguous blocks. Perhaps you could map in chunks and write some intermediate function to decide which to read from/write to?

• Try porting to 64 bit.

Answer 2

As the earlier post suggests, you can suspend every thread in the process while you change the memory mappings. You can use SuspendThread()/ResumeThread() for that. This has the disadvantage that your code has to know about all the other threads and hold thread handles for them.

An alternative is to use the Windows debug API to suspend all threads. If a process has a debugger attached, then every time the process faults, Windows will suspend all of the process's threads until the debugger handles the fault and resumes the process.

Also see this question which is very similar, but phrased differently: Replacing memory mappings atomically on Windows

Answer 3

Have you looked at creating your own private heap via HeapCreate? You could set the heap to your desired buffer size. The only remaining problem is then how to get MapViewOfFileto use your private heap instead of the default heap.

I'd assume that MapViewOfFile internally calls GetProcessHeap to get the default heap and then it requests a contiguous block of memory. You can surround the call to MapViewOfFile with a detour, i.e., you rewire the GetProcessHeap call by overwriting the method in memory effectively inserting a jump to your own code which can return your private heap.

Microsoft has published the Detour Library that I'm not directly familiar with however. I know that detouring is surprisingly common. Security software, virus scanners etc all use such frameworks. It's not pretty, but may work:

HANDLE g_hndPrivateHeap;

HANDLE WINAPI GetProcessHeapImpl() {
    return g_hndPrivateHeap;
}    


struct SDetourGetProcessHeap { // object for exception safety 
   SDetourGetProcessHeap() {
       // put detour in place
   }

   ~SDetourGetProcessHeap() {
       // remove detour again
   }
};


void MapFile() {
    g_hndPrivateHeap = HeapCreate( ... );

    {
        SDetourGetProcessHeap d;
        MapViewOfFile(...);
    }
}

These may also help:

How to replace WinAPI functions calls in the MS VC++ project with my own implementation (name and parameters set are the same)?

How can I hook Windows functions in C/C++?

http://research.microsoft.com/pubs/68568/huntusenixnt99.pdf

Answer 4

You could brute force it; suspend every thread in the process that isn't the one performing the mapping, Unmap/Remap, unsuspend the suspended threads. It ain't elegant, but it's the only way I can think of off-hand to provide the kind of mutual exclusion you need.