I agree with answer from Conrad - yet one more idea to improve performance would be to do reverse lookup i.e. take each element from SortedList and do lookup in other list (of course, I would recommend to have dictionary instead of list for faster lookup).