Why does SqlAzureExecutionStrategy not handle error: 19 - Physical connection is not usable

Question

Full exception:

System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for de         


        

Answer 1

If you have explicitly opened the connection, this is a design decision. Microsoft says:

From the profiler trace we observe that the same connection is used for each query database query. This is by design and as discussed early, i.e. when a connection is explicitly opened by the developer it tells EF not to open/reopen a connection for each command. The series of Audit Login/Logout events to retrieve the customer entity or address entity are not submitted as we saw in Case #1 and #2. This means we cannot implement a retry policy for each individual query like I showed earlier. Since the EntityConnection has been assigned to the ObjectContext, EF takes the position that you really truly want to use one connection for all of your queries within the scope of that context. Retrying a query on an invalid or closed connection can never work, a System.Data.EntityCommandExecutionException will be thrown with an inner SqlException contains the message for the error. (see http://blogs.msdn.com/b/appfabriccat/archive/2010/12/11/sql-azure-and-entity-framework-connection-fault-handling.aspx)

Also, and my apologies if you have already seen this, but Julia at the Data Farm goes into quite of bit of detail for transient errors. I'm not sure if 'Physical connection is not usable' is considered transient - it's not included in the list for System.Data.SqlClient.SqlException in the code for SqlAzureExecutionStrategy - but it might be worth looking at: http://thedatafarm.com/data-access/ef6-connection-resiliency-for-sql-azure-when-does-it-actually-do-its-thing/ (and her follow-up, referenced in the linked article).

I haven't looked deeply into Azure since about 2012, but I hope this helps.

Answer 2

From the profiler trace we observe that the same connection is used for each query database query. This is by design and as discussed early, i.e. when a connection is explicitly opened by the developer it tells EF not to open/reopen a connection for each command.

Well this certainly does not sound like general statement. What profiler trace? Why suppose connection explicitly opened by the developer and handled to the EF? I dont see anything like this in original question (and it is not common practice with EF).

So the questions remain unanswered: Why isn't this handled by the SqlAzureExecutionStrategy? Is it a good idea to write an own DbExecutionStrategy that handles this one?

Since I can see this error in my Azure service from time to time, I decided to test it. Here is my strategy:

public class ExtendedSqlAzureExecutionStrategy : SqlAzureExecutionStrategy
    {
        public ExtendedSqlAzureExecutionStrategy(int maxRetryCount, TimeSpan maxDelay) : base(maxRetryCount, maxDelay) 
        { }

        protected override bool ShouldRetryOn(Exception exception)
        {
            return base.ShouldRetryOn(exception) || IsPhysicalConnectionNotUsableSqlException(exception);
        }

        private bool IsPhysicalConnectionNotUsableSqlException(Exception ex)
        {
            var sqlException = ex as SqlException;
            if (sqlException != null)
            {
                // Enumerate through all errors found in the exception.
                foreach (SqlError err in sqlException.Errors)
                {
                    if (err.Number == 19)
                    {
                        return true;
                    }                    
                }
            }

            return false;
        }
    }

EDIT

Ok, so after some time and logging I can tell that the strategy based on

if (err.Number == 19)

is wrong. Actual SqlException object for this error has ErrorCode = -2146232060 and Number = -1 - I could not find any documentation for those, so I decided not to base strategy on them. For now I am trying trivial check:

public class ExtendedSqlAzureExecutionStrategy : SqlAzureExecutionStrategy
    {
        public ExtendedSqlAzureExecutionStrategy(int maxRetryCount, TimeSpan maxDelay) : base(maxRetryCount, maxDelay) 
        { }

        protected override bool ShouldRetryOn(Exception exception)
        {
            return base.ShouldRetryOn(exception) || IsPhysicalConnectionNotUsableSqlException(exception);
        }

        private bool IsPhysicalConnectionNotUsableSqlException(Exception ex)
        {
            var sqlException = ex as SqlException;
            if (sqlException != null)
            {
                return sqlException.Message.Contains("Physical connection is not usable");
            }

            return false;
        }
    }

EDIT 2:

It works. No more Physical connection is not usable errors at all, and no RetryLimitExceededException, so this error is in fact transient (solvable by retry), so I think it should be included in SqlAzureExecutionStrategy.

Answer 3

My present conclusion is this: the list of errors in SqlAzureExecutionStrategy is nothing more than an educated guess by the Entity Framework team; and it is usually out of date because the EF is not updated in synch with SQL Azure.

I base this conclusion on looking at the history of the list, including a commented out -2 error, and comparing it to other lists in other pieces of code around blogs, the (now obsolete) Transient Error Handling Block, and the ever-changing actual errors that we are personally seeing.

As long as the two teams and codebases are separate, this approach probably won't work. If they made this list of transient errors a property of SQL server that EF could query and use, and made it the SQL Azure teams job to keep it up to date and accurate, then maybe this solution would work.

In the meantime, my conclusion is that the only viable strategy which won't randomly break, is to retry every exception. Sure, some of them are probably useless to retry, but the only sure way is to try.

Answer 4

My problem was with the time of SqlAzureExecutionStrategy, was stablished in 300 secods (5 minutes) and the operations I was doing took more than 5 minutes:

Imports System.Data.Entity
Imports System.Data.Entity.SqlServer

Namespace Models
    Public Class ModelConfiguration
        Inherits DbConfiguration

        Sub New()
            ' https://docs.microsoft.com/es-es/ef/ef6/fundamentals/connection-resiliency/retry-logic

            ' WebConfig
            Dim AzureBBDD = ConfigurationManager.AppSettings("AzureBBDD").Equals("SI")

            ' If AzureBBDD == "SI"
            If (AzureBBDD) Then
                ' WebConfig, here was the problem, low value in seconds...
                Dim AzureBBDDMaxTimeSeconds = Integer.Parse(ConfigurationManager.AppSettings("AzureBBDDMaxTimeSeconds"))
                SetExecutionStrategy("System.Data.SqlClient",
                                 Function()
                                     Return New SqlAzureExecutionStrategy(Integer.MaxValue, TimeSpan.FromSeconds(AzureBBDDMaxTimeSeconds))
                                 End Function)

            End If
        End Sub
    End Class
End Namespace

Answer 5

Physical connection is not usable is a big category of errors, some of them are transient, but some aren't. SqlAzureExecutionStrategy only retries specific errors that are known to be transient.

You should examine all codes contained in .Errors collection to determine whether the action should be retried. Please report if you find specific recurring transient errors that SqlAzureExecutionStrategy misses.

That said, generic errors -1 and -2 can be retried in your custom strategy once you ruled out all possible causes under your control: long running transactions, excessively complex queries, too many concurrently opened connections.