OutOfMemoryException when a lot of memory is available

Question

We have an application that is running on 5 (server) nodes (16 cores, 128 GB Memory each) that loads almost 70 GB data on each machine. This application is distributed and serve

Answer 1

Garbage collection with LargeObjectHeapCompactionMode = CompactOnce may help to fix fragmentation.

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();

Answer 2

Some of the preliminary questions that other users have suggested are cool, but have you considered being lazy and profiling your app?

I can think of Ants profiler from Redgate or dotmemory from JetBrains, links below.

http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

https://www.jetbrains.com/dotmemory/

Answer 3

I suggest that using ADPlus or other tools to get dump of your process when this exception occurs.Using this dump, you can debug your dump file using WinDbg. All of the below commands are taken from blog post Investigating ASP.Net Memory Dumps for Idiots (like Me).

Investigating memory leaks

In order to get a view on memory, we need to use the following command

!dumpheap

"dumpheap" command will give you object counts and memory usage of objects. Then you can investigate which object types uses most of your memory.

!dumpheap -type System.IO.MemoryStream

"dumpheap -type" command will list all of the objects on the heap that are of type MemoryStream. Good thing about WinDbg is you can investigate Unmanaged Memory Leaks: Example 1 and Example2.

Answer 4

Even if there is a memory leak from unmanaged code, if you have 40% memory available you should be able to allocate objects. What I am thinking of is that this is a fragmentation problem not a memory leak.

1- Is the data you are trying to allocate in big or small chunks?

2- Did you try to force the garbage collector (By Calling GC.Collect()) ? garbage collection not only frees memory but compacts it removing fragmentation.