About Microsoft.ServiceBus.Messaging.LeaseLostException

Question

I got an exception of Microsoft.ServiceBus.Messaging.LeaseLostException, in my EventHub processor.

what does this exception mean? What is the possible root cause of this exception?

Following are stack track:

at Microsoft.ServiceBus.Messaging.BlobLeaseManager.d__24.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.ServiceBus.Messaging.BlobLeaseManager.d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n

at MyEventHub.EventProcessor`1.d__6.MoveNext()\r\n\r\nMicrosoft.WindowsAzure.Storage.StorageException : \"The remote server returned an error: (409) Conflict.\":

at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 60\r\n at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.b__3(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 115\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.ServiceBus.Messaging.BlobLeaseManager.d__24.MoveNext()\r\n\r\nSystem.Net.WebException : \"The remote server returned an error: (409) Conflict.\": at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase1 cmd, Exception ex) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\Common\\Shared\\Protocol\\HttpResponseParsers.Common.cs:line 50\r\n at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.<>c__DisplayClass33.<RenewLeaseImpl>b__32(RESTCommand1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 3186\r\n at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 306"

And the only internal exception in my own code is:

at MyEventHub.EventProcessor`1.<< CloseAsync >> d__6.MoveNext()\r\n\r\nMicrosoft.WindowsAzure.Storage.StorageException : \"The remote server returned an error: (409) Conflict.\": ...

And here is the code of CloseAsync:

    public async Task CloseAsync(PartitionContext context, CloseReason reason)
    {
        try
        {
            if (reason == CloseReason.Shutdown)
            {
                await context.CheckpointAsync();
            }
        }
        catch (Exception ex)
        {
            this.HandleException(ex);
        }
        finally
        {
            this.configuration.DecrementOpenedPartitionCount?.Invoke();
        }
    }

MyEventHub is hosted in worker role which deployed 2 instance in Azure.

Answer 1

@Youxu , for your EventHub, how many receivers ( EventProcessorHost ) are configured?. As per my knowledge we can have only one active receiver with certain epoch. If we create a EventHub listener ( with default options) while one listener is already listing to a EventHub, newly created listener gets higher epoch and first listener gets disconnected due to LeaseLostException

Check if you are (accidentally) running more than one receiver for same EventHub concurrently.