I often have to sort a dictionary, consisting of keys & values, by value. For example, I have a hash of words and respective frequencies, that I want to order by frequen
You do not sort entries in the Dictionary. Dictionary class in .NET is implemented as a hashtable - this data structure is not sortable by definition.
If you need to be able to iterate over your collection (by key) - you need to use SortedDictionary, which is implemented as a Binary Search Tree.
In your case, however the source structure is irrelevant, because it is sorted by a different field. You would still need to sort it by frequency and put it in a new collection sorted by the relevant field (frequency). So in this collection the frequencies are keys and words are values. Since many words can have the same frequency (and you are going to use it as a key) you cannot use neither Dictionary nor SortedDictionary (they require unique keys). This leaves you with a SortedList.
I don't understand why you insist on maintaining a link to the original item in your main/first dictionary.
If the objects in your collection had a more complex structure (more fields) and you needed to be able to efficiently access/sort them using several different fields as keys - You would probably need a custom data structure that would consist of the main storage that supports O(1) insertion and removal (LinkedList) and several indexing structures - Dictionaries/SortedDictionaries/SortedLists. These indexes would use one of the fields from your complex class as a key and a pointer/reference to the LinkedListNode in the LinkedList as a value.
You would need to coordinate insertions and removals to keep your indexes in sync with the main collection (LinkedList) and removals would be pretty expensive I'd think. This is similar to how database indexes work - they are fantastic for lookups but they become a burden when you need to perform many insetions and deletions.
All of the above is only justified if you are going to do some look-up heavy processing. If you only need to output them once sorted by frequency then you could just produce a list of (anonymous) tuples:
var dict = new SortedDictionary();
// ToDo: populate dict
var output = dict.OrderBy(e => e.Value).Select(e => new {frequency = e.Value, word = e.Key}).ToList();
foreach (var entry in output)
{
Console.WriteLine("frequency:{0}, word: {1}",entry.frequency,entry.word);
}