Service Fabric Actor or Service Becomes Inaccessible at Random after Upgrading to SDK 2.3.301

Question

After upgrading from Service Fabric SDK 2.0.135 to 2.3.301, we have started encountering situations where a Service Fabric actor or service is inaccessible in spite of showing a

Answer 1

In case it helps anyone we were seeing these timeouts on long running (over 5 minute) operations. Following Suchi's hint about the FabricTransportServiceRemotingProviderAttribute we added the following lines to our SF projects AssemblyInfo.cs to increase the timeout to 1 hour.

[assembly: FabricTransportServiceRemotingProvider(OperationTimeoutInSeconds = 3600)]
[assembly: FabricTransportActorRemotingProvider(OperationTimeoutInSeconds = 3600)]

(Also note if you're using Azure Service Buses the maximum lock time is 5 minutes, so you'll have to implement some lock renewal code to support long running operations)

Answer 2

This issue can happen in 2 scenarios.

1. If your ActorService method processing is taking more than the default timeout, then you need to change OperationTimeout value. By default it is 5 minutes. If you want to change the timeout, you can change it by adding assembly FabricTransportServiceRemotingProviderAttribute in your client assembly.

https://msdn.microsoft.com/en-us/library/microsoft.servicefabric.services.remoting.fabrictransport.fabrictransportserviceremotingproviderattribute.aspx

2. If first scenario is not the case, then you can try below mitigation for a known bug.
   • Specify Port 0 in the Service Manifest for the ActorService endpoint. By default, ActorEndpoint will be listed in ServiceManifest but port won’t be there.

This is how it will look for ActorService after you make change.

<Endpoint Name="Actor1ActorServiceEndpoint" Port="0" />

We are aware of the problem and a fix is on the way.