Heap fragmentation when using byte arrays

Question

I have a C# 4.0 application (single producer/single consumer) which transfers huge amount of data in chunks. Although there\'s no new memory allocation I run out of memory after

Answer 1

You probably ran into the large object heap problem - objects larger than 85,000 bytes are put on the large object heap which is not compacted which can lead to strange out of memory situations. Although apparently the performance in .NET 4 has been improved it's far from perfect. The solution is to basically use your own buffer pool which contains a few statically allocated chunks of memory and reuse those.
There is a whole bunch of questions around that on SO.

Update: Microsoft provides a buffer manager as part of the WCF stack. There is also one on codeproject.