How do I classify documents with SciKitLearn using TfIdfVectorizer?

Question

The following example shows how one can train a classifier with the Sklearn 20 newsgroups data.

>>> from sklearn.feature_extraction.text import TfidfVec         


        

Answer 1

To address questions from comments; The whole basic process of working with tfidf representation in some classification task you should:

1. You fit the vectorizer to your training data and save it in some variable, lets call it tfidf
2. You transform training data (without labels, just text) through data = tfidf.transform(...)
3. You fit the model (classifier) using some_classifier.fit( data, labels ), where labels are in the same order as documnents in data
4. During testing you use tfidf.transform( ... ) on new data, and check the prediction of your model