How to classify documents indexed with lucene

Question

I have classified a set of documents with Lucene (fields: content, category). Each document has it\'s own category, but some of them are labeled as uncategorized. Is there any w

Answer 1

Yes you can use similarity queries such as implemented by the MoreLikeThisQuery class for this kind of things (assuming you have some large text field in the documents for your lucene index). Have a look at the javadoc of the underlying MoreLikeThis class for details on how it works.

To turn your lucene index into a text classifier you have two options:

1. For any new text to classifier, query for the top 10 or 50 most similar documents that have at least one category, sum the category occurrences among those "neighbors" and pick up the top 3 frequent categories among those similar documents (for instance).

2. Alternatively you can index a new set of aggregate documents, one for each category by concatenating (all or a sample of) the text of the documents of this category. Then run similarity query with you input text directly on those "fake" documents.

The first strategy is known in machine learning as k-Nearest Neighbors classification. The second is a hack :)

If you have many categories (say more than 1000) the second option might be better (faster to classify). I have not run any clean performance evaluation though.

You might also find this blog post interesting.

If you want to use Solr, your need to enable the MoreLikeThisHandler and set termVectors=true on the content field.

The sunburnt Solr client for python is able to perform mlt queries. Here is a prototype python classifier that uses Solr for classification using an index of Wikipedia categories:

https://github.com/ogrisel/pignlproc/blob/master/examples/topic-corpus/categorize.py