Azure Redis Cache - Multiple errors TimeoutException: Timeout performing GET {key}

Question

We deployed our application to Azure. It is using the Azure Redis Cache and we are experiencing quite a few timeouts. Namely:

[TimeoutException: Timeout perf         


        

Answer 1

Go to Azure and get a bigger Redis instance. That t

Answer 2

A few points that improved our situation:

Protobuf-net instead of BinaryFormatter

I recommend using protobuf-net as it will reduce the size of values that you want to store in your cache.

public interface ICacheDataSerializer
    {
        byte[] Serialize(object o);
        T Deserialize<T>(byte[] stream);
    }

public class ProtobufNetSerializer : ICacheDataSerializer
    {
        public byte[] Serialize(object o)
        {
            using (var memoryStream = new MemoryStream())
            {
                Serializer.Serialize(memoryStream, o);

                return memoryStream.ToArray();
            }
        }

        public T Deserialize<T>(byte[] stream)
        {
            var memoryStream = new MemoryStream(stream);

            return Serializer.Deserialize<T>(memoryStream);
        }
    }

Implement retry strategy

Implement this RedisCacheTransientErrorDetectionStrategy to handle timeout issues.

using Microsoft.Practices.TransientFaultHandling;

public class RedisCacheTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
    {
        /// <summary>
        /// Custom Redis Transient Error Detenction Strategy must have been implemented to satisfy Redis exceptions.
        /// </summary>
        /// <param name="ex"></param>
        /// <returns></returns>
        public bool IsTransient(Exception ex)
        {
            if (ex == null) return false;

            if (ex is TimeoutException) return true;

            if (ex is RedisServerException) return true;

            if (ex is RedisException) return true;

            if (ex.InnerException != null)
            {
                return IsTransient(ex.InnerException);
            }

            return false;
        }
    }

Instantiate like this:

private readonly RetryPolicy _retryPolicy;

// CODE
var retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(2));
            _retryPolicy = new RetryPolicy<RedisCacheTransientErrorDetectionStrategy>(retryStrategy);

Use like this:

var cachedString = _retryPolicy.ExecuteAction(() => dataCache.StringGet(fullCacheKey));

Review your code to minimize cache calls and values that you are storing in your cache. I reduced lots of errors by storing values more efficiently.

If none of this helps. Move to higher cache (we ended up using C3 instead of C1).

Answer 3

In all of your exception messages, you have something like "in=65536/0". This indicates that 65536 bytes have been received locally (in the kernel's socket buffer on the local machine) but have not yet been processed by StackExchange.Redis. This would point at an issue on the client app side of things that is keeping the system from processing your response. Two possibilities I can think of that could lead to this:

1. Your system does not have enough free thread pool threads to process the data fast enough. If you update to build 1.0.450 or later of StackExchange.Redis, the timeout messages will include statistics for how busy the thread pool is, which will help diagnose this issue. Alternatively, you can take use code like https://github.com/JonCole/SampleCode/blob/master/ThreadPoolMonitor/ThreadPoolLogger.cs to log this same information yourself whenever you get an exception.

2. You are receiving a very large object from the cache and the size of the object is not received within the timeout configured on StackExchange.Redis. When this happens, all calls that were queued onto the same connection behind the large item will timeout when the large item times out.

Anyway, hope this helps.