A process serving application pool 'Classic .NET AppPool' suffered a fatal communication error with the Windows Process Activation Service

Question

I am running an ASP.NET 3.5 web application on IIS 7 on Windows Server 2008.

I received the following error earlier this afternoon:

HTTP Error 503. The service is unavailable.

I looked in the Event logs and discovered that:

A process serving application pool 'Classic .NET AppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3328'. The data field contains the error number.

Here is the full details of that event:

Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          18/09/2009 14:58:31
Event ID:      5011
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      computername
Description:
A process serving application pool 'Classic .NET AppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3328'. The data field contains the error number.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WAS" Guid="{5 .. 0}" EventSourceName="WAS" />
    <EventID Qualifiers="32768">5011</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2009-09-18T13:58:31.000Z" />
    <EventRecordID>23552</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>computername</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="AppPoolID">Classic .NET AppPool</Data>
    <Data Name="ProcessID">3328</Data>
    <Binary>6D000780</Binary>
  </EventData>
</Event>

This resulted in:

Application pool 'Classic .NET AppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

Here is the full details of that event:

Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          18/09/2009 15:03:05
Event ID:      5002
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      computername
Description:
Application pool 'Classic .NET AppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WAS" Guid="{5 .. 0}" EventSourceName="WAS" />
    <EventID Qualifiers="49152">5002</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2009-09-18T14:03:05.000Z" />
    <EventRecordID>23557</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>computername</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="AppPoolID">Classic .NET AppPool</Data>
    <Binary>
    </Binary>
  </EventData>
</Event>

Concluding in the: HTTP Error 503. The service is unavailable.

Would someone please help me to identify how this happened and what I need to do to prevent it from happening in the future.

Answer 1

This one is a pretty nasty one. I recommend getting the IIS Debug Diagnostics tools. You can probably alleviate the symptom by turning off "Rapid Fail Protection" in the properties of your application pool, but that will just mask the problem. What is going on is that your application is experiencing an exception so often/so repeatedly that the service shuts itself off. You'll need to run the diagnostic tool to capture everything going in and out to find the exception that's occuring.

This can be a very laborious task because the tool will gather gigabytes of junk to sift through. If you're lucky, the turning off of the "Rapid Fail Protection" might allow the exception to proceed through to the browser, but my guess is that the server will just end up hanging.

Answer 2

If you are using NServiceBus make sure whatever account the application pool is running under has permissions to read off of the message queue.

NServiceBus runs a background task on application startup that timesout after 10 seconds, causing the w3wp process to be killed if it can't read from the queue. After n (default of 5) w3wp crashes Rapid-Fail Protection kicks in and stops the application pool.

http://blog.jonathanoliver.com/2010/11/nservicebus-w3wp-exe-processes-crashing/

Answer 3

Its most likely caused by an infinite loop.

(I just had the same problem.) So you will just have to look for places in the code that use loops and see if removing one of them stops the error. Once you have found the culprit, then you can try to figure out whats causing it to loop indefinitely.

Answer 4

I just tracked down and fixed the same nasty problem. A couple things that could cause the problem:

• A Server.Transfer() loop
• Calling CurrentThread.Abort() in certain scenarios.. (Like after a Server.Transfer()!)

Fixing those things is what fixed it for us.

Answer 5

I had this problem, the event viewer had "The Module DLL C:\Windows\system32\RpcProxy\RpcProxy.dll failed to load. The data is the error." This post had the steps to correct this: http://forums.iis.net/t/1154189.aspx