Calculate TF-IDF of documents using HBase as the datasource

Question

I want to calculate the TF (Term Frequency) and the IDF (Inverse Document Frequency) of documents that are stored in HBase.

I also want to save the calculated TF in a HBase table, also save the calculated IDF in another HBase table.

Can you guide me through?

I have looked at BayesTfIdfDriver from Mahout 0.4 but I am not getting a head start.

Answer 1

The outline of a solution is pretty straight forward:

1. do a word count over your hbase tables, storing both term frequency and document frequency for each word
2. in your reduce phase aggregate the term frequency and document frequency for each word
3. Given a count of your documents, scan through your aggregated results one more time and calculate the IDF based off of the document frequency.

The wikipedia page on TF-IDF is a good reference to remember the details of the formula: http://en.wikipedia.org/wiki/Tf*idf

Answer 2

If you want to calculate TF, IDF then you need to create intermediate table "TermMatrix" that stores terms with document IDs. Then you can calculate TFIDF by using the TermMatrix table. It is close to real-time but if you want real-time TFIDF output then I would recommend to create "TF", "IDF" tables too.

I wrote a blog about how to calculate TFIDF by using HBase: http://ahikmat.blogspot.kr/2014/07/building-term-matrix-on-hbase.html