WCF is slow when reliable session is ON and with burst async request

Question

For experiments, I created a simple "Hello World" WCF service and client using .NET 4.5 on VS2012. The server is hosted on a console application and use net.tcp binding. I wrote the client code to send a burst of async requests (a total of 700 requests) to the server. Everything went fine until I turned ON the Reliable session feature of the service. After it is ON, the service suddenly ran very slow and it took almost a minute on my machine to complete the 700 requests. I tried to fine tune the Concurrency and Throttling parameters (see below) but it didn't help.

Does anyone know why this happen ? Is there anyway to avoid this ?

The slowness didn't happen if I turned OFF the Reliable session feature, or if I made the service call synchronous. So I think it may relate to the way WCF handles pending requests in WS-ReliableMessaging mode.

EDIT: Also this didn't happen when I chaned netTcpBinding to wsHttpBinding. This is very weird because in this case wsHttpBinding is much faster than netTcpBinding.

EDIT: Running Perfmon.exe on the server side shows that the "Thread Count" gradually increase from 8 to beyond 100 in the above case.

EDIT: Some measured throughput on my PC (local network). See that the performance of case 1 is very sluggish and practically useless.

1. Async + NetTcpBinding/Reliable throughput -> approx. 14 call/s (70 ms/call)
2. Async + WsHttp/Reliable throughput -> 7957 call/s (0.12 ms/call)
3. Sync + NetTcpBinding/Reliable throughtput -> 3986 call/s (0.25 ms/call)

Below are the codes and configuration for the server and client I used in the experiments.

Server:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using System.ServiceModel;
using System.ServiceModel.Description;

[ServiceContract]
public interface IHelloService
{
    [OperationContract(IsOneWay=false)]
    string SayHello(string name);
}

[ServiceBehavior(ConcurrencyMode=ConcurrencyMode.Multiple, InstanceContextMode=InstanceContextMode.PerSession)]
public class HelloService : IHelloService
{
    public string SayHello(string name) {
        String s = string.Format("Hello {0}", name); 
        return s; 
    }
}

namespace WcfServer
{
    class Program
    {
        static void Main(string[] args)
        {
            Uri baseAddress = new Uri("net.tcp://localhost:8080/hello");
            using (ServiceHost host = new ServiceHost(typeof(HelloService), baseAddress)){
                // Open and listen
                host.Open();
                Console.WriteLine("The service is ready at {0}", baseAddress);
                Console.WriteLine("Press <Enter> to stop the service.");
                Console.ReadLine();
                // Close the ServiceHost.
                host.Close();
            }
        }
    }
}

Client:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using System.ServiceModel;
using WcfClient.WcfServer;

namespace WcfClient
{
    class Program
    {
        static async Task PrintNameAsync(HelloServiceClient client, int cnt) {
            string s = await client.SayHelloAsync(string.Format("-- {0} --", cnt));
            Console.WriteLine(s);
        }

        static void Main(string[] args)
        {
            HelloServiceClient client = new HelloServiceClient("HelloService", "net.tcp://10.20.61.13:8080/hello");
            List<Task> tasks = new List<Task>();
            for(int i=0; i < 700; i++){
                Task t = PrintNameAsync(client, i);
                tasks.Add(t);
            }
            Task.WhenAll(tasks).Wait();
            client.Close();
        }
    }
}

Server's App.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
    <system.serviceModel>
        <bindings>
            <netTcpBinding>
                <binding name="HelloServiceBinding">
                    <reliableSession ordered="true" enabled="true" />
                    <security mode="None" />
                </binding>
            </netTcpBinding>
        </bindings>
        <behaviors>
            <serviceBehaviors>
                <behavior name="HelloServiceBehavior">
                    <serviceMetadata policyVersion="Policy15" />
                    <serviceDebug includeExceptionDetailInFaults="true" />
                    <serviceThrottling maxConcurrentCalls="1000" maxConcurrentSessions="1000"
                        maxConcurrentInstances="1000" />
                </behavior>
            </serviceBehaviors>
        </behaviors>
        <services>
            <service behaviorConfiguration="HelloServiceBehavior" name="HelloService">
                <endpoint address="net.tcp://localhost:8080/hello" binding="netTcpBinding"
                    bindingConfiguration="HelloServiceBinding" name="HelloService" contract="IHelloService" />
                <endpoint address="mex" binding="mexTcpBinding" contract="IMetadataExchange" />
            </service>
        </services>
    </system.serviceModel>
</configuration>

Client's App.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
    <system.serviceModel>
        <bindings>
            <netTcpBinding>
                <binding name="HelloServiceBinding" sendTimeout="00:01:00">
                    <reliableSession enabled="true" />
                    <security mode="None" />
                </binding>
            </netTcpBinding>
        </bindings>
        <client>
            <endpoint address="net.tcp://localhost:8080/hello" binding="netTcpBinding"
                bindingConfiguration="HelloServiceBinding" contract="WcfServer.IHelloService"
                name="HelloService">
            </endpoint>
        </client>
    </system.serviceModel>
</configuration>

Answer 1

You are sending messages Async, but you have ordered="true" on the ReliableSessionBindingElement. This doesn't make sense. Set ordered to false, since it makes more sense for your scenario. ReliableMessaging will cause a performance hit because it adds to every response message a SequenceAcknowledgement. It also adds overhead of CreateSequence/CreateSequenceResponse and CloseSequence/CloseSequenceResponse, then TerminateSequence/TerminateSequenceResponse message exchanges at the beginning and end of your session.

Answer 2

Found a partial workaround for the problem from the below links:

• http://blogs.msdn.com/b/endpoint/archive/2011/05/04/wcf-scales-up-slowly-with-bursts-of-work.aspx
• http://support.microsoft.com/kb/2538826

With the workaround (using the WorkerThreadPoolBehavior), the measured throughputs are as follows:

1. Async + NetTcpBinding/Reliable throughput -> 474 call/s (2.1 ms/call) ... improved but not satisfactorily
2. Async + WsHttp/Reliable throughput -> 7856 call/s (0.13 ms/call) ... no change
3. Sync + NetTcpBinding/Reliable throughtput -> 2110 call/s 0.47 ms/call) ... degraded

Note that the case 1 above is improved significantly from 70 ms/call. However, it still lags from case 2. And for case 3, introducing WorkerThreadPool behavior cause performance degradation from 0.25 ms/call to 0.47 ms/call.

Answer 3

Look at this...

"Setting MaxTransferWindowSize

Reliable sessions in Windows Communication Foundation (WCF) use a transfer window to hold messages on the client and service. The configurable property MaxTransferWindowSize indicates how many messages the transfer window can hold.

On the sender, this indicates how many messages the transfer window can hold while waiting for acknowledgements; on the receiver it indicates how many messages to buffer for the service..."

Source "MSDN: Best Practices for Reliable Sessions": http://msdn.microsoft.com/en-us/library/ms733795.aspx