How do I classify documents with SciKitLearn using TfIdfVectorizer?

Question

The following example shows how one can train a classifier with the Sklearn 20 newsgroups data.

>>> from sklearn.feature_extraction.text import TfidfVec         


        

Answer 1

In general, for sklearn the flow is:

1. Convert your string data to numeric values usinf some vectorizer for e.g. TfIDF,count etcs
2. fit and transform
3. Pass it to train/fit of your choice of classifier.

You did not mention your data format but if it is csv file with some rows,flow could be:

1. Read each row of text
2. Pre process, like remove the stop words etc.
3. raw_data_list = [row1,row2,rown...]
4. vectorizer = TfidfVectorizer()
5. x_transformed = vectorizer.fit_transform(raw_data_list)
6. x_transformed can be passed to fit/train function of classifiers.

And once you have trained classifier you can call predict for new data. Remeber to convert new data to same format as data on which you trained by using above used and fitted vectorizer before passing it to classif.predict.