问题
I am getting familiar with NLTK and text categorization by Jacob Perkins's book "Python Text Processing with NLTK 2.0 Cookbook".
My corpus documents/texts each consists of a paragraph of text, so each of them is in a separate line of file, not in a separate file. The number of such these paragraphs/lines are about 2 millions. Therefore there are about 2 million on machine learning instances.
Each line in my file (a paragraph of text - a combination of domain title, description, keywords), that is a subject of feature extraction: tokenization, etc. to make it an instance for a machine learning algorithm.
I have two files like that with all the positives and negavives.
How can I load it to CategorizedCorpusReader? Is it possible?
I tried other solutions before, like scikit, and finally picked NLTK hoping for an easier point to start with a result.
回答1:
Assuming that you have two files:
file_pos.txt, file_neg.txt
from nltk.corpus.reader import CategorizedCorpusReader
reader = CategorizedCorpusReader('/path/to/corpora/', \
r'file_.*\.txt', \
cat_pattern=r'file_(\w+)\.txt')
After this, you can apply the usual Corpus functions to it like:
>>> reader.categories()
['neg', 'pos']
>>> reader.fileids(categories=['neg'])
['file_neg.txt']
As well as tagged_sents, tagged_words, etc.
You might enjoy this tutorial about creating a custom corpora: https://www.packtpub.com/books/content/python-text-processing-nltk-20-creating-custom-corpora
来源：https://stackoverflow.com/questions/27538853/need-to-set-categorized-corpus-reader-in-nltk-and-python-corpus-texts-in-one-fi