Near Duplicate Detection in Data Streams

Question

I am currently working on a streaming API that generates a lot of textual content. As expected, the API gives out a lot of duplicates and we also have a business requirement to

Answer 1

http://micvog.com/2013/09/08/storm-first-story-detection/ has some nice implementation notes