Locking pattern for proper use of .NET MemoryCache

Question

I assume this code has concurrency issues:

const string CacheKey = \"CacheKey\";
static string GetCachedData()
{
    string expensiveString =null;
    if (Me         


        

Answer 1

There is an open source library [disclaimer: that I wrote]: LazyCache that IMO covers your requirement with two lines of code:

IAppCache cache = new CachingService();
var cachedResults = cache.GetOrAdd("CacheKey", 
  () => SomeHeavyAndExpensiveCalculation());

It has built in locking by default so the cacheable method will only execute once per cache miss, and it uses a lambda so you can do "get or add" in one go. It defaults to 20 minutes sliding expiration.

There's even a NuGet package ;)

Answer 2

I assume this code has concurrency issues:

Actually, it's quite possibly fine, though with a possible improvement.

Now, in general the pattern where we have multiple threads setting a shared value on first use, to not lock on the value being obtained and set can be:

1. Disastrous - other code will assume only one instance exists.
2. Disastrous - the code that obtains the instance is not can only tolerate one (or perhaps a certain small number) concurrent operations.
3. Disastrous - the means of storage is not thread-safe (e.g. have two threads adding to a dictionary and you can get all sorts of nasty errors).
4. Sub-optimal - the overall performance is worse than if locking had ensured only one thread did the work of obtaining the value.
5. Optimal - the cost of having multiple threads do redundant work is less than the cost of preventing it, especially since that can only happen during a relatively brief period.

However, considering here that MemoryCache may evict entries then:

1. If it's disastrous to have more than one instance then MemoryCache is the wrong approach.
2. If you must prevent simultaneous creation, you should do so at the point of creation.
3. MemoryCache is thread-safe in terms of access to that object, so that is not a concern here.

Both of these possibilities have to be thought about of course, though the only time having two instances of the same string existing can be a problem is if you're doing very particular optimisations that don't apply here*.

So, we're left with the possibilities:

1. It is cheaper to avoid the cost of duplicate calls to SomeHeavyAndExpensiveCalculation().
2. It is cheaper not to avoid the cost of duplicate calls to SomeHeavyAndExpensiveCalculation().

And working that out can be difficult (indeed, the sort of thing where it's worth profiling rather than assuming you can work it out). It's worth considering here though that most obvious ways of locking on insert will prevent all additions to the cache, including those that are unrelated.

This means that if we had 50 threads trying to set 50 different values, then we'll have to make all 50 threads wait on each other, even though they weren't even going to do the same calculation.

As such, you're probably better off with the code you have, than with code that avoids the race-condition, and if the race-condition is a problem, you quite likely either need to handle that somewhere else, or need a different caching strategy than one that expels old entries†.

The one thing I would change is I'd replace the call to Set() with one to AddOrGetExisting(). From the above it should be clear that it probably isn't necessary, but it would allow the newly obtained item to be collected, reducing overall memory use and allowing a higher ratio of low generation to high generation collections.

So yeah, you could use double-locking to prevent concurrency, but either the concurrency isn't actually a problem, or your storing the values in the wrong way, or double-locking on the store would not be the best way to solve it.

*If you know only one each of a set of strings exists, you can optimise equality comparisons, which is about the only time having two copies of a string can be incorrect rather than just sub-optimal, but you'd want to be doing very different types of caching for that to make sense. E.g. the sort XmlReader does internally.

†Quite likely either one that stores indefinitely, or one that makes use of weak references so it will only expel entries if there are no existing uses.

Answer 3

public interface ILazyCacheProvider : IAppCache
{
    /// <summary>
    /// Get data loaded - after allways throw cached result (even when data is older then needed) but very fast!
    /// </summary>
    /// <param name="key"></param>
    /// <param name="getData"></param>
    /// <param name="slidingExpiration"></param>
    /// <typeparam name="T"></typeparam>
    /// <returns></returns>
    T GetOrAddPermanent<T>(string key, Func<T> getData, TimeSpan slidingExpiration);
}

/// <summary>
/// Initialize LazyCache in runtime
/// </summary>
public class LazzyCacheProvider: CachingService, ILazyCacheProvider
{
    private readonly Logger _logger = LogManager.GetLogger("MemCashe");
    private readonly Hashtable _hash = new Hashtable();
    private readonly List<string>  _reloader = new List<string>();
    private readonly ConcurrentDictionary<string, DateTime> _lastLoad = new ConcurrentDictionary<string, DateTime>();  


    T ILazyCacheProvider.GetOrAddPermanent<T>(string dataKey, Func<T> getData, TimeSpan slidingExpiration)
    {
        var currentPrincipal = Thread.CurrentPrincipal;
        if (!ObjectCache.Contains(dataKey) && !_hash.Contains(dataKey))
        {
            _hash[dataKey] = null;
            _logger.Debug($"{dataKey} - first start");
            _lastLoad[dataKey] = DateTime.Now;
            _hash[dataKey] = ((object)GetOrAdd(dataKey, getData, slidingExpiration)).CloneObject();
            _lastLoad[dataKey] = DateTime.Now;
           _logger.Debug($"{dataKey} - first");
        }
        else
        {
            if ((!ObjectCache.Contains(dataKey) || _lastLoad[dataKey].AddMinutes(slidingExpiration.Minutes) < DateTime.Now) && _hash[dataKey] != null)
                Task.Run(() =>
                {
                    if (_reloader.Contains(dataKey)) return;
                    lock (_reloader)
                    {
                        if (ObjectCache.Contains(dataKey))
                        {
                            if(_lastLoad[dataKey].AddMinutes(slidingExpiration.Minutes) > DateTime.Now)
                                return;
                            _lastLoad[dataKey] = DateTime.Now;
                            Remove(dataKey);
                        }
                        _reloader.Add(dataKey);
                        Thread.CurrentPrincipal = currentPrincipal;
                        _logger.Debug($"{dataKey} - reload start");
                        _hash[dataKey] = ((object)GetOrAdd(dataKey, getData, slidingExpiration)).CloneObject();
                        _logger.Debug($"{dataKey} - reload");
                        _reloader.Remove(dataKey);
                    }
                });
        }
        if (_hash[dataKey] != null) return (T) (_hash[dataKey]);

        _logger.Debug($"{dataKey} - dummy start");
        var data = GetOrAdd(dataKey, getData, slidingExpiration);
        _logger.Debug($"{dataKey} - dummy");
        return (T)((object)data).CloneObject();
    }
}

Answer 4

To avoid the global lock, you can use SingletonCache to implement one lock per key, without exploding memory usage (the lock objects are removed when no longer referenced, and acquire/release is thread safe guaranteeing that only 1 instance is ever in use via compare and swap).

Using it looks like this:

SingletonCache<string, object> keyLocks = new SingletonCache<string, object>();

const string CacheKey = "CacheKey";
static string GetCachedData()
{
    string expensiveString =null;
    if (MemoryCache.Default.Contains(CacheKey))
    {
        return MemoryCache.Default[CacheKey] as string;
    }

    // double checked lock
    using (var lifetime = keyLocks.Acquire(url))
    {
        lock (lifetime.Value)
        {
           if (MemoryCache.Default.Contains(CacheKey))
           {
              return MemoryCache.Default[CacheKey] as string;
           }

           cacheItemPolicy cip = new CacheItemPolicy()
           {
              AbsoluteExpiration = new DateTimeOffset(DateTime.Now.AddMinutes(20))
           };
           expensiveString = SomeHeavyAndExpensiveCalculation();
           MemoryCache.Default.Set(CacheKey, expensiveString, cip);
           return expensiveString;
        }
    }      
}

Code is here on GitHub: https://github.com/bitfaster/BitFaster.Caching

Install-Package BitFaster.Caching

There is also an LRU implementation that is lighter weight than MemoryCache, and has several advantages - faster concurrent reads and writes, bounded size, no background thread, internal perf counters etc. (disclaimer, I wrote it).

Answer 5

It is difficult to choose which one is better; lock or ReaderWriterLockSlim. You need real world statistics of read and write numbers and ratios etc.

But if you believe using "lock" is the correct way. Then here is a different solution for different needs. I also include the Allan Xu's solution in the code. Because both can be needed for different needs.

Here are the requirements, driving me to this solution:

1. You don't want to or cannot supply the 'GetData' function for some reason. Perhaps the 'GetData' function is located in some other class with a heavy constructor and you do not want to even create an instance till ensuring it is unescapable.
2. You need to access the same cached data from different locations/tiers of the application. And those different locations don't have access to same locker object.
3. You don't have a constant cache key. For example; need of caching some data with the sessionId cache key.

Code:

using System;
using System.Runtime.Caching;
using System.Collections.Concurrent;
using System.Collections.Generic;

namespace CachePoc
{
    class Program
    {
        static object everoneUseThisLockObject4CacheXYZ = new object();
        const string CacheXYZ = "CacheXYZ";
        static object everoneUseThisLockObject4CacheABC = new object();
        const string CacheABC = "CacheABC";

        static void Main(string[] args)
        {
            //Allan Xu's usage
            string xyzData = MemoryCacheHelper.GetCachedDataOrAdd<string>(CacheXYZ, everoneUseThisLockObject4CacheXYZ, 20, SomeHeavyAndExpensiveXYZCalculation);
            string abcData = MemoryCacheHelper.GetCachedDataOrAdd<string>(CacheABC, everoneUseThisLockObject4CacheXYZ, 20, SomeHeavyAndExpensiveXYZCalculation);

            //My usage
            string sessionId = System.Web.HttpContext.Current.Session["CurrentUser.SessionId"].ToString();
            string yvz = MemoryCacheHelper.GetCachedData<string>(sessionId);
            if (string.IsNullOrWhiteSpace(yvz))
            {
                object locker = MemoryCacheHelper.GetLocker(sessionId);
                lock (locker)
                {
                    yvz = MemoryCacheHelper.GetCachedData<string>(sessionId);
                    if (string.IsNullOrWhiteSpace(yvz))
                    {
                        DatabaseRepositoryWithHeavyConstructorOverHead dbRepo = new DatabaseRepositoryWithHeavyConstructorOverHead();
                        yvz = dbRepo.GetDataExpensiveDataForSession(sessionId);
                        MemoryCacheHelper.AddDataToCache(sessionId, yvz, 5);
                    }
                }
            }
        }


        private static string SomeHeavyAndExpensiveXYZCalculation() { return "Expensive"; }
        private static string SomeHeavyAndExpensiveABCCalculation() { return "Expensive"; }

        public static class MemoryCacheHelper
        {
            //Allan Xu's solution
            public static T GetCachedDataOrAdd<T>(string cacheKey, object cacheLock, int minutesToExpire, Func<T> GetData) where T : class
            {
                //Returns null if the string does not exist, prevents a race condition where the cache invalidates between the contains check and the retreival.
                T cachedData = MemoryCache.Default.Get(cacheKey, null) as T;

                if (cachedData != null)
                    return cachedData;

                lock (cacheLock)
                {
                    //Check to see if anyone wrote to the cache while we where waiting our turn to write the new value.
                    cachedData = MemoryCache.Default.Get(cacheKey, null) as T;

                    if (cachedData != null)
                        return cachedData;

                    cachedData = GetData();
                    MemoryCache.Default.Set(cacheKey, cachedData, DateTime.Now.AddMinutes(minutesToExpire));
                    return cachedData;
                }
            }

            #region "My Solution"

            readonly static ConcurrentDictionary<string, object> Lockers = new ConcurrentDictionary<string, object>();
            public static object GetLocker(string cacheKey)
            {
                CleanupLockers();

                return Lockers.GetOrAdd(cacheKey, item => (cacheKey, new object()));
            }

            public static T GetCachedData<T>(string cacheKey) where T : class
            {
                CleanupLockers();

                T cachedData = MemoryCache.Default.Get(cacheKey) as T;
                return cachedData;
            }

            public static void AddDataToCache(string cacheKey, object value, int cacheTimePolicyMinutes)
            {
                CleanupLockers();

                MemoryCache.Default.Add(cacheKey, value, DateTimeOffset.Now.AddMinutes(cacheTimePolicyMinutes));
            }

            static DateTimeOffset lastCleanUpTime = DateTimeOffset.MinValue;
            static void CleanupLockers()
            {
                if (DateTimeOffset.Now.Subtract(lastCleanUpTime).TotalMinutes > 1)
                {
                    lock (Lockers)//maybe a better locker is needed?
                    {
                        try//bypass exceptions
                        {
                            List<string> lockersToRemove = new List<string>();
                            foreach (var locker in Lockers)
                            {
                                if (!MemoryCache.Default.Contains(locker.Key))
                                    lockersToRemove.Add(locker.Key);
                            }

                            object dummy;
                            foreach (string lockerKey in lockersToRemove)
                                Lockers.TryRemove(lockerKey, out dummy);

                            lastCleanUpTime = DateTimeOffset.Now;
                        }
                        catch (Exception)
                        { }
                    }
                }

            }
            #endregion
        }
    }

    class DatabaseRepositoryWithHeavyConstructorOverHead
    {
        internal string GetDataExpensiveDataForSession(string sessionId)
        {
            return "Expensive data from database";
        }
    }

}

Answer 6

This is my 2nd iteration of the code. Because MemoryCache is thread safe you don't need to lock on the initial read, you can just read and if the cache returns null then do the lock check to see if you need to create the string. It greatly simplifies the code.

const string CacheKey = "CacheKey";
static readonly object cacheLock = new object();
private static string GetCachedData()
{

    //Returns null if the string does not exist, prevents a race condition where the cache invalidates between the contains check and the retreival.
    var cachedString = MemoryCache.Default.Get(CacheKey, null) as string;

    if (cachedString != null)
    {
        return cachedString;
    }

    lock (cacheLock)
    {
        //Check to see if anyone wrote to the cache while we where waiting our turn to write the new value.
        cachedString = MemoryCache.Default.Get(CacheKey, null) as string;

        if (cachedString != null)
        {
            return cachedString;
        }

        //The value still did not exist so we now write it in to the cache.
        var expensiveString = SomeHeavyAndExpensiveCalculation();
        CacheItemPolicy cip = new CacheItemPolicy()
                              {
                                  AbsoluteExpiration = new DateTimeOffset(DateTime.Now.AddMinutes(20))
                              };
        MemoryCache.Default.Set(CacheKey, expensiveString, cip);
        return expensiveString;
    }
}

---

EDIT: The below code is unnecessary but I wanted to leave it to show the original method. It may be useful to future visitors who are using a different collection that has thread safe reads but non-thread safe writes (almost all of classes under the System.Collections namespace is like that).

Here is how I would do it using ReaderWriterLockSlim to protect access. You need to do a kind of "Double Checked Locking" to see if anyone else created the cached item while we where waiting to to take the lock.

const string CacheKey = "CacheKey";
static readonly ReaderWriterLockSlim cacheLock = new ReaderWriterLockSlim();
static string GetCachedData()
{
    //First we do a read lock to see if it already exists, this allows multiple readers at the same time.
    cacheLock.EnterReadLock();
    try
    {
        //Returns null if the string does not exist, prevents a race condition where the cache invalidates between the contains check and the retreival.
        var cachedString = MemoryCache.Default.Get(CacheKey, null) as string;

        if (cachedString != null)
        {
            return cachedString;
        }
    }
    finally
    {
        cacheLock.ExitReadLock();
    }

    //Only one UpgradeableReadLock can exist at one time, but it can co-exist with many ReadLocks
    cacheLock.EnterUpgradeableReadLock();
    try
    {
        //We need to check again to see if the string was created while we where waiting to enter the EnterUpgradeableReadLock
        var cachedString = MemoryCache.Default.Get(CacheKey, null) as string;

        if (cachedString != null)
        {
            return cachedString;
        }

        //The entry still does not exist so we need to create it and enter the write lock
        var expensiveString = SomeHeavyAndExpensiveCalculation();
        cacheLock.EnterWriteLock(); //This will block till all the Readers flush.
        try
        {
            CacheItemPolicy cip = new CacheItemPolicy()
            {
                AbsoluteExpiration = new DateTimeOffset(DateTime.Now.AddMinutes(20))
            };
            MemoryCache.Default.Set(CacheKey, expensiveString, cip);
            return expensiveString;
        }
        finally 
        {
            cacheLock.ExitWriteLock();
        }
    }
    finally
    {
        cacheLock.ExitUpgradeableReadLock();
    }
}