Checking for duplicates in a List of Objects C#

Question

I am looking for a really fast way to check for duplicates in a list of objects.

I was thinking of simply looping through the list and doing a manual comparison th

Answer 1

If any duplicate occurs throws exception. Dictionary checks keys by itself. this is the easiest way.

try
{
  dupList.ToDictionary(a=>new {a.checkThis,a.checkThat});
}
catch{
 //message: list items is not uniqe
}

Answer 2

For in memory objects I always use the Distinct LINQ method adding a comparer to the solution.

public class dupeCheckee
{
     public string checkThis { get; set; }
     public string checkThat { get; set; }

     dupeCheckee(string val, string val2)
     {
         checkThis = val;
         checkThat = val2;
     }

     public class Comparer : IEqualityComparer<dupeCheckee>
     {
         public bool Equals(dupeCheckee x, dupeCheckee y)
         {
             if (x == null || y == null)
                 return false;

             return x.CheckThis == y.CheckThis && x.CheckThat == y.CheckThat;
         }

         public int GetHashCode(dupeCheckee obj)
         {
             if (obj == null)
                 return 0;

             return (obj.CheckThis == null ? 0 : obj.CheckThis.GetHashCode()) ^
                 (obj.CheckThat == null ? 0 : obj.CheckThat.GetHashCode());
         }
     }
}

Now we can call

List<dupeCheckee> dupList = new List<dupeCheckee>();
dupList.Add(new dupeCheckee("test1", "value1"));
dupList.Add(new dupeCheckee("test2", "value1"));
dupList.Add(new dupeCheckee("test3", "value1"));
dupList.Add(new dupeCheckee("test1", "value1"));//dupe
dupList.Add(new dupeCheckee("test2", "value1"));//dupe... 
dupList.Add(new dupeCheckee("test4", "value1"));
dupList.Add(new dupeCheckee("test5", "value1"));
dupList.Add(new dupeCheckee("test1", "value2"));//not dupe

var distinct = dupList.Distinct(dupeCheckee.Comparer);

Answer 3

I like using this for knowing when there are any duplicates at all. Lets say you had a string and wanted to know if there was any duplicate letters. This is what I use.

string text = "this is some text";

var hasDupes = text.GroupBy(x => x).Any(grp => grp.Count() > 1);

If you wanted to know how many duplicates there are no matter what the duplicates are, use this.

var totalDupeItems = text.GroupBy(x => x).Count(grp =>  grp.Count() > 1);

So for example, "this is some text" has this...

total of letter t: 3

total of letter i: 2

total of letter s: 3

total of letter e: 2

So variable totalDupeItems would equal 4. There are 4 different kinds of duplicates.

If you wanted to get the total amount of dupe items no matter what the dupes are, then use this.

var totalDupes = letters.GroupBy(x => x).Where(grp => grp.Count() > 1).Sum(grp => grp.Count());

So the variable totalDupes would be 10. This is the total duplicate items of each dupe type added together.

Answer 4

There was huge amount of working solutions, but I think that next solution will be more transparent and easy to understand, then all above:

var hasDuplicatedEntries = ListWithPossibleDuplicates
                                   .GroupBy(YourGroupingExpression)
                                   .Any(e => e.Count() > 1);
if(hasDuplicatedEntries)
{
   // Do what ever you want in case when list contains duplicates 
}

Answer 5

Do a select distinct with linq, e.g. How can I do SELECT UNIQUE with LINQ?

And then compare counts of the distinct results with the non-distinct results. That will give you a boolean saying if the list has doubles.

Also, you could try using a Dictionary, which will guarantee the key is unique.

Answer 6

You need to reference System.Linq (e.g. using System.Linq)

then you can do

var dupes = dupList.GroupBy(x => new {x.checkThis, x.checkThat})
                   .Where(x => x.Skip(1).Any());

This will give you groups with all the duplicates

The test for duplicates would then be

var hasDupes = dupList.GroupBy(x => new {x.checkThis, x.checkThat})
                   .Where(x => x.Skip(1).Any()).Any();

or even call ToList() or ToArray() to force the calculation of the result and then you can both check for dupes and examine them.

eg..

var dupes = dupList.GroupBy(x => new {x.checkThis, x.checkThat})
                   .Where(x => x.Skip(1).Any()).ToArray();
if (dupes.Any()) {
  foreach (var dupeList in dupes) {
    Console.WriteLine(string.Format("checkThis={0},checkThat={1} has {2} duplicates",
                      duplist.Key.checkThis, 
                      duplist.Key.checkThat,
                      duplist.Count() - 1));
  }

}

Alternatively

var dupes = dupList.Select((x, i) => new { index = i, value = x})
                   .GroupBy(x => new {x.value.checkThis, x.value.checkThat})
                   .Where(x => x.Skip(1).Any());

Which give you the groups which each item per group stores the original index in a property index and the item in the property value