Large Object Heap and String Objects coming from a queue

时光毁灭记忆、已成空白 提交于 2019-12-03 11:25:01
Justin

Your problem appears to be due to memory allocation on the large object heap - the large object heap is not compacted and so can be a source of fragmentation. There is a good article here that goes into more detail including some debugging steps that you can follow to confirm that fragmentation of the large object heap is happening:

Large Object Heap Uncovered

You appear to have two three solutions:

  1. Alter your application to perform processing on chunks / shorter strings, where each chunk is smaller than 85,000 bytes - this avoids the allocation of large objects.
  2. Alter your application to allocate a few large chunks of memory up-front and re-use those chunks by copying new messages into the allocated memory instead. See Heap fragmentation when using byte arrays.
  3. Leave things as they are - As long as you don't experience out of memory exceptions and the application isn't interfering with other applications running on the system you should probably leave things as they are.

Its important here to understand the distinction between virtual memory and physical memory - even though the process is using a large amount of virtual memory, if the number of objects allocated is relatively low then it cam be that the physical memory use of that process is low (the un-used memory is paged to disk) meaning little impact on other processes on the system. You may also find that the "VM Hoarding" option helps - read "Large Object Heap Uncovered" article for more information.

Either change involves changing your application to perform either some or all of its processing using byte arrays and short substrings instead of a single large string - how difficult this is going to be for you will depend on what sort of processing it is that you are doing.

When there is fragmentation on the LOH, it means that there are allocated objects on it. If you can affort the delay, you can once in a while wait till all currently running tasks are finished and call GC.Collect(). When there are no referenced large objects, they will all be collected, effectively removing the fragmentation of the LOH. Of course this only works if (allmost) all large objects are unreferenced.

Also, moving to a 64 bit OS might also help, since out of memory due to fragmentation is much less likely to be a problem on 64 bits systems, because the virtual space is almost unlimited.

Perhaps you could create a pool of string objects which you can use whilst processing the work then return back once you've finished.

Once a large object has been created in the LOH, it can't be removed (AFAIK), so if you can't avoid creating these objects then the best plan is to reuse them.

If you can change the protocol at both ends, then reducing your 'Contents' string into a set of smaller ones (<80k each) should stop them from being stored in the LOH.

How about using String.Intern(...) to eliminate duplicates references. It has a performance penalty, but depending on your strings it might have an impact.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!