WCF Silverlight client getting 404 not found response for poll message

Question

Eventually WCF duplex Silverlight 4 client start getting 404 Not Found errors for poll messages, immediately after poll was send from WCF service to Silverlight client, sometimes this happens for second poll sometimes connectivity works hours or even days, but mostly fails on first minutes.

! And what is interesting the issue is like known Silverlight 4 bug when using MaxMessagesPerPollduplex mode and solution is described here and here but I'm using SingleMessagePerPoll mode. ANyway I tried using ClientStack as suggested but nothing changed.

General flow:

1. SL client executes WCF service method, received response
2. Then immediately SL client start sending poll messages to service and then getting exception for second or N-s poll message

   System.Net.WebException: The remote server returned an error: NotFound

3. Fiddler show only empty 404 response for a poll message
4. Then client Channel Faulted event raised

I'm trying to reconnect SL client after such a fault, single reconnect retry flow:

1. Handle Faulted event
2. Unsubscribe all channel events like Closed/Closing/Opened/Opening
3. Close channel in a right way using try { close } catch { abort }
4. All below in a new thread poll thread:(I foudn this works slightly stable - see this article)
5. Wait 45-70 seconds
6. Using the same DuplexChannelFactory<T> instance create a new channel, subscribe to all channel events just for logging purposes
7. Execute WCF service method

After 1-10 retries (~1-10 minutes) client eventually connect to a server and continue normal polling.

In WCF service log I see it get all cleint request, processed without any exception, so it seems something happens on Silverlight Client side.

General info:

• .NET Framework 4.0
• PollingDuplex
• Async WCF methods
• IIS 6.0 hosted WCF service
• Silverligth 4 client
• Client OS: Windows XP SP2
• Server OS: Windows 2003 R2 SP2
• NTLM Authentication
• DuplexMode: SingleMessagePerPoll
• There is an other WCF service which does Request/Reply before my service start working, it not uses Duplex connectivity
• On SL client service I'm logging everything into the UI so see all events flow and have time for each particular event
• No errors in IIS logs, server event logs

Client:

var binaryBinding = new BinaryMessageEncodingBindingElement();
binaryBinding.ReaderQuotas.MaxStringContentLength = int.MaxValue;

var httpbindingElement = new HttpTransportBindingElement
{
    MaxReceivedMessageSize = 131072
};

var pollingDuplexBindingElement = new PollingDuplexBindingElement
{
    ClientPollTimeout = new TimeSpan(0, 0, 1, 30),
    InactivityTimeout = new TimeSpan(0, 8, 0, 0),
};


_binding = new CustomBinding(
           pollingDuplexBindingElement,
           binaryBinding,
           httpbindingElement)
       {
           SendTimeout = new TimeSpan(0, 0, 0, 45),
           CloseTimeout = new TimeSpan(0, 0, 0, 25),
           ReceiveTimeout = new TimeSpan(0, 8, 0, 0),
           OpenTimeout = new TimeSpan(0, 0, 0, 45)
       };


httpbindingElement.AuthenticationScheme = AuthenticationSchemes.Negotiate;
var endpoint = new EndpointAddress(_endpointAddress);
_channelFactory = new DuplexChannelFactory<TWebService>(
                       new InstanceContext(instanceOfClientServiceClass), 
                       _binding, 
                       endpoint);


// then this factory used to create a new channels
// Also for a new channel I'm setting OpTimeout
var contextChannel = newChannel as IContextChannel;
if (contextChannel != null)
{
   contextChannel.OperationTimeout = TimeSpan.FromSeconds(45);
}

Server:

• WCF, PerSession, multithreaded
• Everything is thread safe
• No server service exceptions while execution
• A lot of logging so I see what is going on on a service
• All WCF Traces are enabled with switchValue All, nothing suspicious

<binding name="customName"
             sendTimeout="00:01:00"
             receiveTimeout="08:00:00"
             openTimeout="00:01:00"
             closeTimeout="00:00:35">
     <pollingDuplex
         inactivityTimeout="08:00:00"
         serverPollTimeout="00:01:00" />
         <binaryMessageEncoding />
           <httpTransport authenticationScheme="Ntlm"
                          maxReceivedMessageSize="131072">              
         </httpTransport>
</binding>

<behavior name="customBehavior">
        <dataContractSerializer maxItemsInObjectGraph="2147483647" />
        <serviceDebug includeExceptionDetailInFaults="true" />
        <serviceThrottling
             maxConcurrentCalls = "500"
             maxConcurrentSessions = "500"
             maxConcurrentInstances = "500" />
</behavior>

Answer 1

If everything seems to be working fine, then perhaps this is a network infrastructure/configuration issue (e.g. dns configuration). Do you get the same issue when running locally or using ip address instead of host name?

Similar issue may also occur if you have several bindings configured on Site in IIS (see here for details: http://blogs.msdn.com/b/rampo/archive/2008/02/11/how-can-wcf-support-multiple-iis-binding-specified-per-site.aspx)

Another thing is how do you communicated from server to clients. If you iterate clients in a loop and call callback methods one by one, then you may get timeouts which would show as 404s. Callback generally should be invoked on background threads instead (one per client).

Depending on how you do the communication, this may also be caused by the deadlock (when UI thread is involved in sending/receiving messages/callbacks to and from service).

Answer 2

While investigating issue described on this StackOverflow post Static constructor called twice for PerSession WCF service I found that Polling Duplex start working stable when I switched IIS configuration for underlying AppPool to use single worker process rather than 2 as was specified before. I'm not sure why 2 was set before since I do not own this server, but anyway this is what I have now - multiple Silverlight clients launched on the same machine works stable and polls polls polls and no 404 errors, all clients reconnects in 1 attempt after IIS restarts and recycles...

See Performance Application Pool Settings for more details

TL;DR: When IIS hosted WCF resides in AppPool which has more than one worker process - polling duplex becomes unstable. So in case of high load IIS started second process and start creating WCF service instances in second process as well, so I ran into situation when client session was created in one process but it seems poll sometimes reached an other process whcih is not aware of current connection/session so start refusing such messages and entire connection fault.

So Polling Duplex by design is not scalable across multiple processes in scope of a single IIS server and AppPool, in other words if you have more than 1 worker process - this is WebGarden environment and duplex is not scalable across web farms and gardens