Fastest way to Remove Duplicate Value from a list<> by lambda

Question

what is fastest way to remove duplicate values from a list. Assume List longs = new List { 1, 2, 3, 4, 3, 2, 5 }; So I am interesting in

Answer 1

You can use this extension method for enumerables containing more complex types:

IEnumerable<Foo> distinctList = sourceList.DistinctBy(x => x.FooName);

public static IEnumerable<TSource> DistinctBy<TSource, TKey>(
    this IEnumerable<TSource> source,
    Func<TSource, TKey> keySelector)
{
    var knownKeys = new HashSet<TKey>();
    return source.Where(element => knownKeys.Add(keySelector(element)));
}

Answer 2

There is Distinct() method. it should works.

List<long> longs = new List<long> { 1, 2, 3, 4, 3, 2, 5 };
var distinctList = longs.Distinct().ToList();

Answer 3

A simple intuitive implementation

public static List<PointF> RemoveDuplicates(List<PointF> listPoints)
{
    List<PointF> result = new List<PointF>();

    for (int i = 0; i < listPoints.Count; i++)
    {
        if (!result.Contains(listPoints[i]))
            result.Add(listPoints[i]);
    }

    return result;
}

Answer 4

If you want to stick with the original List instead of creating a new one, you can something similar to what the Distinct() extension method does internally, i.e. use a HashSet to check for uniqueness:

HashSet<long> set = new HashSet<long>(longs.Count);
longs.RemoveAll(x => !set.Add(x));

The List class provides this convenient RemoveAll(predicate) method that drops all elements not satisfying the condition specified by the predicate. The predicate is a delegate taking a parameter of the list's element type and returning a bool value. The HashSet's Add() method returns true only if the set doesn't contain the item yet. Thus by removing any items from the list that can't be added to the set you effectively remove all duplicates.

Answer 5

In-place:

    public static void DistinctValues<T>(List<T> list)
    {
        list.Sort();

        int src = 0;
        int dst = 0;
        while (src < list.Count)
        {
            var val = list[src];
            list[dst] = val;

            ++dst;
            while (++src < list.Count && list[src].Equals(val)) ;
        }
        if (dst < list.Count)
        {
            list.RemoveRange(dst, list.Count - dst);
        }
    }

Answer 6

The easiest way to get a new list would be:

List<long> unique = longs.Distinct().ToList();

Is that good enough for you, or do you need to mutate the existing list? The latter is significantly more long-winded.

Note that Distinct() isn't guaranteed to preserve the original order, but in the current implementation it will - and that's the most natural implementation. See my Edulinq blog post about Distinct() for more information.

If you don't need it to be a List<long>, you could just keep it as:

IEnumerable<long> unique = longs.Distinct();

At this point it will go through the de-duping each time you iterate over unique though. Whether that's good or not will depend on your requirements.