Check for duplicates in Blocking collection [duplicate]

Answer 1

You will have to implement your own IProducerConsumerCollection that behaves like a set (e.g. no duplicates allowed). Here is a simplistic version that use a critical section (C# lock) to make it thread-safe. For high concurrency scenarios you may be able to improve performance by using a class like SpinWait the same way as ConcurrentQueue does.

public class ProducerConsumerSet : IProducerConsumerCollection {

  readonly object gate = new object();

  readonly Queue queue = new Queue();

  readonly HashSet hashSet = new HashSet();

  public void CopyTo(T[] array, int index) {
    if (array == null)
      throw new ArgumentNullException("array");
    if (index < 0)
      throw new ArgumentOutOfRangeException("index");
    lock (gate)
      queue.CopyTo(array, index);
  }

  public bool TryAdd(T item) {
    lock (gate) {
      if (hashSet.Contains(item))
        return false;
      queue.Enqueue(item);
      hashSet.Add(item);
      return true;
    }
  }

  public bool TryTake(out T item) {
    lock (gate) {
      if (queue.Count == 0) {
        item = default(T);
        return false;
      }
      item = queue.Dequeue();
      hashSet.Remove(item);
      return true;
    }
  }

  public T[] ToArray() {
    lock (gate)
      return queue.ToArray();
  }

  public void CopyTo(Array array, int index) {
    if (array == null)
      throw new ArgumentNullException("array");
    lock (gate)
      ((ICollection) queue).CopyTo(array, index);
  }

  public int Count {
    get { return queue.Count; }
  }

  public object SyncRoot {
    get { return gate; }
  }

  public bool IsSynchronized {
    get { return true; }
  }

  public IEnumerator GetEnumerator() {
    List list = null;
    lock (gate)
      list = queue.ToList();
    return list.GetEnumerator();
  }

  IEnumerator IEnumerable.GetEnumerator() {
    return GetEnumerator();
  }

}

If required you can elaborate on this class to customize equality by supplying an optional IEqualityComparer that then is used to initialize the HashSet.

The IProducerConsumerCollection.Add methods returns false when there is an attempt to insert a duplicate item. This results in an InvalidOperationException thrown by the BlockingCollection.Add method so you will probably have to wrap the code to add an item into something like this:

bool AddItem(BlockingCollection blockingCollection, T item) {
  try {
    blockingCollection.Add(item);
    return true;
  }
  catch (InvalidOperationException) {
    return false;
  }
}

Note that if you add items to a collection that has been completed you will also get an InvalidOperationException and you will have to examine the exception message to determine the underlying reason for the exception.