Can address space be recycled for multiple calls to MapViewOfFileEx without chance of failure?

Question

Consider a complex, memory hungry, multi threaded application running within a 32bit address space on windows XP.

Certain operations require n large buffers of fixed

Answer 1

Have you looked at creating your own private heap via HeapCreate? You could set the heap to your desired buffer size. The only remaining problem is then how to get MapViewOfFileto use your private heap instead of the default heap.

I'd assume that MapViewOfFile internally calls GetProcessHeap to get the default heap and then it requests a contiguous block of memory. You can surround the call to MapViewOfFile with a detour, i.e., you rewire the GetProcessHeap call by overwriting the method in memory effectively inserting a jump to your own code which can return your private heap.

Microsoft has published the Detour Library that I'm not directly familiar with however. I know that detouring is surprisingly common. Security software, virus scanners etc all use such frameworks. It's not pretty, but may work:

HANDLE g_hndPrivateHeap;

HANDLE WINAPI GetProcessHeapImpl() {
    return g_hndPrivateHeap;
}    


struct SDetourGetProcessHeap { // object for exception safety 
   SDetourGetProcessHeap() {
       // put detour in place
   }

   ~SDetourGetProcessHeap() {
       // remove detour again
   }
};


void MapFile() {
    g_hndPrivateHeap = HeapCreate( ... );

    {
        SDetourGetProcessHeap d;
        MapViewOfFile(...);
    }
}

These may also help:

How to replace WinAPI functions calls in the MS VC++ project with my own implementation (name and parameters set are the same)?

How can I hook Windows functions in C/C++?

http://research.microsoft.com/pubs/68568/huntusenixnt99.pdf