What would be the cause of this worker process crash?

Question

We experienced a worker process crash in our ASP.NET 2.0 application.

Here is the report that was dumped when the worker process crashed:

Version=1
Event         


        

Answer 1

The clue to this problem is in the Exception Code:

Sig[6].Name=Exception Code
Sig[6].Value=e053534f      <====== this is the clue

This exception code indicates that there was a "soft stack overflow" raised by the CLR. Please see the commentary at the end of this MSDN Library article:

StackOverflowException Class

What you'll need to do is install the Debug Diagnostic Toolkit and configure it to dump your worker process whenever this crash happens.

Once you've captured a dump of the process you'll need to load this into WinDBG and use SOS to narrow down the root cause.

There are several great articles on how to analyse a worker process crash dump:

.NET Debugging Demos - Information and setup instructions - Tess Ferrandez

Diagnosing Tricky ASP.Net Production Issues with DebugDiag and winDbg

System.StackOverflowException with IronPython 2.6.0 and 2.6.1

The articles by Tess Ferrandez are great and I learned to master WinDBG and SOS from her labs.

Answer 2

To add to Kev's excellent answer -

1. Right Click > My Computer > Properties > Advanced Tab > click the Settings button under Start up and Recovery > choose "Complete Memory Dump" and specify a location with enough HDD space equal to the amount of RAM in the server.

2. Once the crash has reoccurred and the memory dump is successful. Install WinDBG, open up WinDBG, File > Open > load the memory dump (.dmp)

3. Then load SOS (eg .loadby sos mscorwks)

4. !clrstack

This will churn out a huge stack trace that will most likely identify a recursive function or some huge allocation as Brian Grunkemeyer - MSFT said in the link Kev provided.

Your probably going to have to put a limit on the depth of the recursive calls... or something similar.

Good luck! You can always call Microsoft PSS if you need help with the diagnosis. Remember to download the Public Symbols from Microsoft before you issue !analyze and !clrstack - in order to see method names. If you need to see method bodies in Microsoft DLLs in a dump file you need the private symbols (only avail to Microsoft engineers).