64 bit large mallocs

Question

What are the reasons a malloc() would fail, especially in 64 bit?

My specific problem is trying to malloc a huge 10GB chunk of RAM on a 64 bit system. The machine ha

Answer 1

It is most likely fragmentation. For simplicity, let's use an example.

The memory consists of a single 12kb module. This memory is organised into 1kb blocks in the MMU. So, you have 12 x 1kb blocks. Your OS uses 100 bytes but this is basically the code that manages the page tables. So, you cannot swap it out. Then, your apps all use 100 bytes each.

Now, with just your OS and your application running (200 bytes), you would already be using 200 bytes of memory (occupying 2kb blocks). Leaving exactly 10kb available for malloc().

Now, you started by malloc() a couple of buffers - A (900 byte), B (200 byte). Then, you free up A. Now, you have 9.8kb free (non-contiguous). So, you try to malloc() C (9kb). Suddenly, you fail.

You have 8.9k contiguous at the tail end and 0.9k at the front end. You cannot re-map the first block to the end because B stretches over the first 1k and the second 1k blocks.

You can still malloc() a single 8kb block.

Granted, this example is a little contrived, but hope it helps.

Answer 2

Just a guess here, but malloc allocates contiguous memory and you may not have a sufficiently large contiguous section on your heap. Here's a few things I would try;

Where a 20GB malloc fails, do four 5GB mallocs succeed? If so, it is a contiguous space issue.

Have you checked your compiler switches for anything that limits total heap size, or largest heap block size?

Have you tried writing a program that declares a static variable of the required size? If this works you could implement your own heap with big mallocs in that space.

Answer 3

Have you tried using VirtualAlloc() and VirtualFree() directly? This may help isolate the problem.

• You'll be bypassing the C runtime heap and the NT heap.
• You can reserve virtual address space and then commit it. This will tell you which operation fails.

If the virtual address space reservation fails (even though it shouldn't, judging from what you've said), Sysinternals VMMap may help explain why. Turn on "Show free regions" to look at how the free virtual address space is fragmented.