News clustering

Question

How does Google News and Techmeme cluster news items that are similar? Are there any well know algorithm that is used to achieve this?

Appreciate your help.

Than

Answer 1

There's a few different ways to do it. The standard is to do a "bag of words" analysis (weighted TF-IDF), and then do cosine similarity and k-means.

I've had success with this paper: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=4289851

The great thing about it is: 1) It's incremental, which is great for news. With standard k-means, you need to have the entire data set. With news, you usually have articles arriving over time. Incremental algorithms solve that. 2) It's phrase-based. So it relies on phrases rather than just words.

Recently, there have been techniques that use semantic meaning instead of words (for instance, by extracting Wikipedia or DBPedia concepts from each article, and using that instead of just words).

Answer 2

One fairly common way to cluster text based on content is to use Principle Component Analysis on the word vectors (a vector of n dimensions where each possible word represents one dimension and the magnitude in each direction, for each vector, is the number occurrences of the word in that particular article), followed by just a simple clustering such as K-Means.

Answer 3

The algorithmic basis is agglomerative clustering or something similar. But there are a number of heuristics on top of that. For example, the vector space is surely comprised of words and phrases (word n-grams). Limiting the search in a strict time period is also very important. And identifying names, and weighing more the title and the paragraph headings are also key parts.

On a tangentially related note. If you are interested in finding near-duplicate articles then there are a number of easier to implement approaches, such as the one described here