Large Object Heap and String Objects coming from a queue

Question

I have a windows console app that is supposed to run without restarts for days and months. The app retrieves \"work\" from an MSMQ and process it. There are 30 threads that proc

Answer 1

Your problem appears to be due to memory allocation on the large object heap - the large object heap is not compacted and so can be a source of fragmentation. There is a good article here that goes into more detail including some debugging steps that you can follow to confirm that fragmentation of the large object heap is happening:

Large Object Heap Uncovered

You appear to have two three solutions:

1. Alter your application to perform processing on chunks / shorter strings, where each chunk is smaller than 85,000 bytes - this avoids the allocation of large objects.
2. Alter your application to allocate a few large chunks of memory up-front and re-use those chunks by copying new messages into the allocated memory instead. See Heap fragmentation when using byte arrays.
3. Leave things as they are - As long as you don't experience out of memory exceptions and the application isn't interfering with other applications running on the system you should probably leave things as they are.

Its important here to understand the distinction between virtual memory and physical memory - even though the process is using a large amount of virtual memory, if the number of objects allocated is relatively low then it cam be that the physical memory use of that process is low (the un-used memory is paged to disk) meaning little impact on other processes on the system. You may also find that the "VM Hoarding" option helps - read "Large Object Heap Uncovered" article for more information.

Either change involves changing your application to perform either some or all of its processing using byte arrays and short substrings instead of a single large string - how difficult this is going to be for you will depend on what sort of processing it is that you are doing.