I have just come across the MemoryCache which is new in .NET 4.
I get that it can be useful if you want to:
I think you nailed the two compelling reasons :-)
The MemoryCache has an eviction strategy, so that it can throw out entries that are no longer needed or for that you do not have enough memory anymore.
A Dictionary will not "lose contents".
Update: MemoryCache is thread-safe and has methods such as AddOrGetExisting. With a Dictionary, you'd have to synchronize access yourself (or use ConcurrentDictionary).