CHECK UPDATE 3 below I found out the issue I ran into is related to a known serious problem with c# string comparers for .Net 4.0, 4.0 client and 4.5, that
Could it be related to .Net Framework 4/4.5? I have adapted your example for .Net 3.5 like this:
var words = ReadFile("hamlet.txt");
//...
private static string[] ReadFile(string path)
{
List<string> lines = new List<string>();
using (StreamReader sr = new StreamReader(path))
{
string text = sr.ReadToEnd();
lines.Add(text);
}
return lines.SelectMany(l => l.Split(new[] { ' ', '\t' }, StringSplitOptions.RemoveEmptyEntries).Select(w => w.Trim()))
.Where(w => !(w.ToCharArray().All(c => c == ' ')))
.ToArray();
}
And both comparers work fine on XP using .Net 3.5.