Thanks to all. Rob, your solution was pretty close to what I ended up with.
Based on your insights I did some more searching and came across Marc Gravel's answer here Dynamic LINQ OrderBy on IEnumerable (second post).
It added dynamic's as an additional bonus.
