Finding memory leak

Question

I have a web application that I wrote using a lot of different 3rd party components, a CMS and of course my code. For some reason I get out of memory exception.

Answer 1

Since it appears you can reproduce your issue, sometimes the simplest approach is to remove things you think might be the root cause, and test again (unless you need a lot of time to see if it grows a lot or not?)

At some point the bug will stop happening, and you'll know what bit of code was responsible.

However, depending on your code base, it's not always easy to remove code and still have something that you can test (i.e. without crashing the application).

Answer 2

When it comes to fixing memory leaks there are two steps.

1. Finding where the memory leak is.
2. Fixing the memory leak.

Normally the first step is tricky. So, I would recommend to go ahead with ANTS memory profiler and first find what exact instances are growing.

Profiling an ASP.NET application on IIS

In your question you have shown the class list result which includes even classes with system namespace. To clean up the noise, you can select "Show Only Classes with source" option.

Then follow the follwing steps.

1. Take a base snapshot before you perform any operations.
2. Perform the operations you suspect that have memory leaks.
3. Take couple of more snapshots until you get a fairly stable snapshot.
4. Compare the last snapshot with the base and see what instances are growing.

Answer 3

You mentioned "I've added a space in web.config and suddenly my IIS process started to grow to more than a GB in 30 minutes". In which web.config tag have you appended this space. What part of your code uses it. Is that part of your code unable to handle some exception without causing memory leak. Use PerfView (Dump GC heap) to analyze the dump. This can exactly tell, the object of what kind is taking up this much of memory. In older versions of .net, it would be objects on large object heap (big arrays) or open and mis-handled db connections and files.

https://channel9.msdn.com/Series/PerfView-Tutorial/Tutorial-10-Investigating-NET-Heap-Memory-Leaks-Part1-Collecting-the-data

https://channel9.msdn.com/Series/PerfView-Tutorial/Tutorial-11-Investigating-NET-Heap-Memory-Leaks-Part2-Analyzing-the-data

Answer 4

In your analysis, I see that the .net analysis is not done correctly. Are you doing analysis in the same machine where you have captured the memory dump ?

For debugdiag to work correctly, you have to have the same version of .net framework (of the application) installed on the analyzing machine as well.

Also Please do not do native memory leak dump like this ,unless it is not figured out unmanaged leak.From the analysis of yours, it looks like this is managed leak.

When you changed the web.config file,it causes an Application domain unload and reload

Let's do step by step

1. Using DebugDiag(capture consecutive hang dumps)
   • Launch DebugDiag Collection and Go to the Processes tab
   • Start your stress test
   • check the memory usage and once it reaches say 1 GB,capture a hang dump
     • right click on the w3wp.exe process
     • choose the option Create Full Memory dump
   • Capture another dump at 2 GB and one at 3.5 GB
   • You should have dumps captured in the folder C:\ProgramFiles\DebugDiag\Logs\Misc\
   • Right click on the dump file and choose the option Analyse .NET Memory Issue

Now compare the analysis of each dump file(1 GB,2 GB, 3.5 GB), it should tell you which .NET objects are increasing and not getting garbage collected.

In the memory analysis, you should see CLR Information****,.NET GC Heap Information ,Most memory Consuming .NET Objects etc like below. This will come if your .net symbols are correctly identified by debugdiag analysis

CLR Information
 CLR version = 4.6.1648.0
 Microsoft.Diagnostics.Runtime version = 0.9.2.0
.NET GC Heap Information
Number of GC Heaps: 4 
Heap Size 0x4001ce8 (67,116,264) 
Heap Size 0x3d5cca0 (64,343,200) 
Heap Size 0x3f8b0d0 (66,629,840) 
Heap Size 0x3ccb0d0 (63,746,256) 
GC Heap Size 249.71 MBytes  
Total Commit Size  249 MB 
Total Reserved Size    17158 MB 

40 most memory consuming .NET object types

System.Char[]   193.01 MBytes    (12450 objects )
Free      45.21 MBytes    (4760 objects )
System.String      1.56 MBytes    (18072 objects ) 
==============trimmed out =======================

DebugDaig Automated analysis should give following

1. **Errors or Warnings **- pay close attention to warning or errors displayed by debugdiag analysis on the top of the report.
2. .NET GC Heap Information - Total Commit Size - This will roughly equal to your .net memory usage.
3. 40 most memory consuming .NET object types -This can be used to compare against your memory increase in consecutive dumps. This will tell you which objects are causing issue.Sometimes some objects you are not using at all will be coming and might be coming from some third party library.Or you will see objects which you yourself have created
4. Top Objects in the Finalizer queue - This will give you any clue if your finalizer may be blocked .Objects.Some similar issues are discussed here and here
5. Objects on the Large Object Heap - This causes memory fragmentation and large object heap contaisn objects which are more than 85K in size.
6. Size of Cache, Datatables,Application domains,Dynamic assemblies etc. It's not a good idea to have large number of application domain in one process

Please note that sometimes the debugdiag automated analysis cannot figure out the root cause and needs manual analysis using windbg. DebugDiag analysis,please refer this video.

Hope this helps!