Error deleting from Azure Cloud Table - ResourceNotFound

Question

I\'m having an intermittent problem deleting objects from an azure table. It only affects about 1% of my attempts, and if the same call is made again later then it works fin

Answer 1

Problem solved, and I'll explain what I found in case it benefits anyone else.

It was due to threading (as eluded to by Smarx), the fact that I was thinking like a SQL developer, and what I would consider some rather odd/unexpected behaviour by the Azure code and a real lack of in-depth examples!

So, to solve it I simplified the problem as much as possible.

I created a table that contained one entity, PartitionKey 'a', RowKey '1'.

I created a console app that selected 'a' from the table, deleted it, changed it to 'b' and re-inserted it.

I ran the code on a loop, thousands of times, and it all worked fine.

I then moved the code into a thread, and launched two threads. Immediately I was hit with errors and lost the entity somewhere between a delete and an insert.

Problem 1: The two threads can both select the entity at the same, both can check to see if it exists, then both can try and delete it. Only one will succeed when deleting it. So you need to catch the error, check that the error contains "ResourceNotFound", and if so trust that the object is really gone and continue as normal.

Problem 2: The tableContext remembers the last failed action, so the thread where the delete failed will throw another error on SaveChangesWithRetries after you call AddObject. So you need to use a new tableContext for the AddObject

Problem 3: Both threads have a chance at adding the entity, but only one will succeed. Even if both threads check to see if the objects exists before adding it, they could both think that it does NOT exist and both attempt to add it. So for simplicity, let both threads try and add it, one will succeed and one will throw an "EntityAlreadyExists" error. Just catch this error and continue.

Here is my working code for this simple example, I modified it for my more complex example in the original question and now receive no errors at all.

    //method for shifting an entity backwards and forwards between two partitions, a and b
    private static void Shift(int threadNumber)
    {
        Console.WriteLine("Launching shift thread " + threadNumber);

        //set up access to the tables
        _storageAccount = CloudStorageAccount.Parse(_connectionString);
        _tableClient = new CloudTableClient(_storageAccount.TableEndpoint.AbsoluteUri, _storageAccount.Credentials);
        _tableClient.RetryPolicy = RetryPolicies.Retry(_retryAmount, TimeSpan.FromSeconds(_retrySeconds));

        int lowerLimit = threadNumber * _limit;
        int upperLimit = (threadNumber + 1) * _limit;

        for (int i = lowerLimit; i < upperLimit; i++)
        {
            try
            {
                TableServiceContext tableServiceContextDelete = _tableClient.GetDataServiceContext();
                tableServiceContextDelete.IgnoreResourceNotFoundException = true;

                string partitionKey = "a";

                if (i % 2 == 1)
                {
                    partitionKey = "b";
                }

                //find the object with this partition key
                var results = from table in tableServiceContextDelete.CreateQuery<TableEntity>(_tableName)
                              where table.PartitionKey == partitionKey
                                && table.RowKey == "1"
                              select table;

                TableEntity tableEntity = results.FirstOrDefault();

                //shallow copy it
                if (tableEntity != null)
                {
                    TableEntity tableEntityShallowCopy = new TableEntity(tableEntity);

                    if (tableEntityShallowCopy.PartitionKey == "a")
                    {
                        tableEntityShallowCopy.PartitionKey = "b";
                    }
                    else
                    {
                        tableEntityShallowCopy.PartitionKey = "a";
                    }

                    //delete original
                    try
                    {
                        tableServiceContextDelete.Detach(tableEntity);
                        tableServiceContextDelete.AttachTo(_tableName, tableEntity, "*");
                        tableServiceContextDelete.DeleteObject(tableEntity);
                        tableServiceContextDelete.SaveChangesWithRetries();
                        Console.WriteLine("Thread " + threadNumber + ". Successfully deleted. PK: " + tableEntity.PartitionKey);
                    }
                    catch (Exception ex1)
                    {
                        if (ex1.InnerException.Message.Contains("ResourceNotFound"))
                        {
                            //trying to delete an object that's already been deleted so just continue
                        }
                        else
                        {
                            Console.WriteLine("Thread " + threadNumber + ". WTF?! Unexpected error during delete. Code: " + ex1.InnerException.Message);
                        }
                    }

                    //move into new partition (a or b depending on where it was taken from)
                    TableServiceContext tableServiceContextAdd = _tableClient.GetDataServiceContext();
                    tableServiceContextAdd.IgnoreResourceNotFoundException = true;

                    try
                    {
                        tableServiceContextAdd.AddObject(_tableName, tableEntityShallowCopy);
                        tableServiceContextAdd.SaveChangesWithRetries();
                        Console.WriteLine("Thread " + threadNumber + ". Successfully inserted. PK: " + tableEntityShallowCopy.PartitionKey);
                    }
                    catch (Exception ex1)
                    {
                        if (ex1.InnerException.Message.Contains("EntityAlreadyExists"))
                        {
                            //trying to add an object that already exists, so continue as normal
                        }
                        else
                        {
                            Console.WriteLine("Thread " + threadNumber + ". WTF?! Unexpected error during add. Code: " + ex1.InnerException.Message);
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error shifting: " + i + ". Error: " + ex.Message + ". " + ex.InnerException.Message + ". " + ex.StackTrace);
            }
        }

        Console.WriteLine("Done shifting");
    }

I'm sure there are much nicer ways of doing this, but due to the lack of good examples I'm just going with something that works for me!

Thanks

Answer 2

By default MergeOption.AppendOnly is used by your tableServiceContext.MergeOption, this means that Azure Table Storage will be tracking your table items. You should detach the item before deleting it, like this:

tableServiceContext.Detach(messageUnread);
tableServiceContext.AttachTo(messageTableName, messageUnread, "*");
tableServiceContext.DeleteObject(messageUnread);
tableServiceContext.SaveChangesWithRetries();

This should get rid of any item tracking issues.

Answer 3

Is this code running in more than one role instance or in more than one app? (Maybe the entity is being deleted by another process between the time when you read it from the table and the time when you try to delete it.)