How to protect application pools from session serialization exceptions?

Question

We're using an Out-of-Process Session Provider (ScaleOut) for an ASP.NET application and we've noticed that when an object that's not correctly setup for de-serialization inadvertently makes its way into session it will eventually cause the entire process to terminate.

Reproducing and handling this scenario is where it gets even more interesting.

The exception that terminates the process is raised in AnyStaObjectsInSessionState whose implementation is pretty straightforward:

internal static bool AnyStaObjectsInSessionState(HttpSessionState session)
{
    if (session != null)
    {
        int count = session.Count;
        for (int i = 0; i < count; i++)
        {
            object obj2 = session[i];
            if (((obj2 != null) && (obj2.GetType().FullName == "System.__ComObject"))
                && (UnsafeNativeMethods.AspCompatIsApartmentComponent(obj2) != 0))
            {
                return true;
            }
        }
    }
    return false;
}

Here's the stack trace that shows how exceptions here terminate the process:

An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1/ROOT

Process ID: 4208

Exception: System.Runtime.Serialization.SerializationException

Message: The constructor to deserialize an object of type 'Lucene.Net.QueryParsers.ParseException' was not found.

StackTrace:    at System.Runtime.Serialization.ObjectManager.CompleteISerializableObject(Object obj, SerializationInfo info, StreamingContext context)
   at System.Runtime.Serialization.ObjectManager.FixupSpecialObject(ObjectHolder holder)
   at System.Runtime.Serialization.ObjectManager.DoFixups()
   at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Web.Util.AltSerialization.ReadValueFromStream(BinaryReader reader)
   at System.Web.SessionState.SessionStateItemCollection.ReadValueFromStreamWithAssert()
   at System.Web.SessionState.SessionStateItemCollection.DeserializeItem(String name, Boolean check)
   at System.Web.SessionState.SessionStateItemCollection.DeserializeItem(Int32 index)
   at System.Web.SessionState.SessionStateItemCollection.get_Item(Int32 index)
   at System.Web.SessionState.HttpSessionStateContainer.get_Item(Int32 index)
   at System.Web.Util.AspCompatApplicationStep.AnyStaObjectsInSessionState(HttpSessionState session)
   at System.Web.HttpApplicationFactory.FireSessionOnEnd(HttpSessionState session, Object eventSource, EventArgs eventArgs)
   at System.Web.SessionState.SessionOnEndTargetWorkItem.RaiseOnEndCallback()
   at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)

InnerException: System.Runtime.Serialization.SerializationException

Message: The constructor to deserialize an object of type 'Lucene.Net.QueryParsers.ParseException' was not found.

StackTrace:    at System.Runtime.Serialization.ObjectManager.GetConstructor(Type t, Type[] ctorParams)
   at System.Runtime.Serialization.ObjectManager.CompleteISerializableObject(Object obj, SerializationInfo info, StreamingContext context)

We would like to understand two things:

1. When does FireSessionOnEnd fire for an out-of-process provider and, more importantly, how can we mimic this in a development environment that's not under load? I've experimented with lowered session timeouts (set to a minute), manually invoking Abandon(), and manually invoking GC.Collect(), all to no avail.

2. Can we trap errors that happen at this step to protect the app pool? The exceptions raised here are logged w/ Source=ASP.NET 2.0.50727.0 and don't reach the application error handlers in global.asax. What can we do to guard against this scenario, even after appropriate checks & balances are applied to session-bound objects?

Any insights would be appreciated.

Answer 1

We were able to resolve this issue with the help of SOSS technical support – they were tremendously helpful – here are the details:

• Upon session expiry, SOSS raises an expiration event in its client libraries, which in turn are responsible for firing Session_End in Global.asax (N.B: ScaleOut load balances expiration events across clients, so the web server that created the session may not necesarily receive its expiration event - this is critical for trying to reproduce these issues).
• Because this happens outside the context of a request, the exception is unhandled and kills the app pool;
• It’s an extremely uncommon scenario but one that they will nonetheless address in upcoming maintenance releases;

• The remedies are as follows:

  1. Fix the System.Exception-derived type (that’s serializable but not unserializable);

  2. Remove Session_End events in Global.asax or disable the expiration events (max_event_retries set to 0 in soss_params.txt);

  3. In these scenarios, it’s likely that the user encounters a SerializationException on one of their requests, meaning it reaches Application_Error; here you can clear the session keys (must clear all of them) or abandon the session outright;

  4. Subscribe to AppDomain.UnhandledException to be notified of unhandled exceptions, should they occur (no recourse here, just logging); they can also be disabled via legacyUnhandledExceptionPolicy (not recommended);

Answer 2

Can we trap errors that happen at this step to protect the app pool? The exceptions raised here are logged w/ Source=ASP.NET 2.0.50727.0 and don't reach the application error handlers in global.asax. What can we do to guard against this scenario, even after appropriate checks & balances are applied to session-bound objects?

I don't know if this will work, but you can give it a shot

Answer 3

I fixed this by simply removing the SessionEnd methods completely. It isn't enough to remove the contents of the methods as Asp.net searches for the existence of the method using reflection and then runs the offending code.