document-classification

I'm trying to make an SMS SPAM classifier using the WEKA library. I have a CSV file with "label" and "text" headings. When I use the code below, it creates an ARFF file with two attributes: @attribute label {ham,spam} @attribute text {'Go until jurong point','Ok lar...', etc.} Currently, it seems that the text attribute is formatted as a nominal attribute with each message's text as a value. But I need the text attribute to be a String attribute, not a list of all of the text from all instances. Having the text attribute as a String will allow me to use the StringToWordVector filter for

I am trying to learn the Stanford NLP Classifier and would like to work on the problem of document classification. Can anyone suggest a place where I can find a working example? I was also looking at the Open NLP libraries and was able to find many working examples, like http://tharindu-rusira.blogspot.com/2013/12/opennlp-text-classifier.html So, as we can see here, it is quite easy to figure out what's going on and create a small working prototype. However, I can't find a simple example for stanford NLP which will show me How to specify training data for a classifier. How to train a model.

I'm trying to implement the naive Bayes classifier for sentiment analysis. I plan to use the TF-IDF weighting measure. I'm just a little stuck now. NB generally uses the word(feature) frequency to find the maximum likelihood. So how do I introduce the TF-IDF weighting measure in naive Bayes? You can visit the following blog shows in detail how do you calculate TFIDF. You use the TF-IDF weights as features/predictors in your statistical model. I suggest to use either gensim [1]or scikit-learn [2] to compute the weights, which you then pass to your Naive Bayes fitting procedure. The scikit-learn

I have been blowing my brains out over the past 2-3 weeks on this problem. I have a multi-label (not multi-class) problem where each sample can belong to several of the labels. I have around 4.5 million text documents as training data and around 1 million as test data. The labels are around 35K. I am using scikit-learn . For feature extraction I was previously using TfidfVectorizer which didn't scale at all, now I am using HashVectorizer which is better but not that scalable given the number of documents that I have. vect = HashingVectorizer(strip_accents='ascii', analyzer='word', stop_words=

I am using document-term vectors to represent a collection of document. I use TF*IDF to calculate the term weight for each document vector. Then I could use this matrix to train a model for document classification. I am looking forward to classify new document in future. But in order to classify it, I need to turn the document into a document-term vector first, and the vector should be composed of TF*IDF values, too. My question is, how could I calculate the TF*IDF with just a single document? As far as I understand, TF can be calculated based on a single document itself, but the IDF can only

My objective is to [semi]automatically assign texts to different categories. There's a set of user defined categories and a set of texts for each category. The ideal algorithm should be able to learn from a human-defined classification and then classify new texts automatically. Can anybody suggest such an algorithm and perhaps .NET library that implements ше? Ralph M. Rickenbach Doing this is not trivial. Obviously you can build a dictionary that maps certain keywords to categories. Just finding a keyword would suggest a certain category. Yet, in natural language text, the keywords would

I am using libsvm (svmutils) from python for a classification task. The classifier is exact. However, I am getting output like this: * optimization finished, #iter = 75 nu = 0.000021 obj = -0.024330, rho = 0.563710 nSV = 26, nBSV = 0 Total nSV = 26 * optimization finished, #iter = 66 nu = 0.000030 obj = -0.035536, rho = -0.500676 nSV = 21, nBSV = 0 Total nSV = 21 * optimization finished, #iter = 78 nu = 0.000029 obj = -0.033921, rho = -0.543311 nSV = 23, nBSV = 0 Total nSV = 23 * optimization finished, #iter = 90 nu = 0.000030 obj = -0.035333, rho = -0.634721 nSV = 23, nBSV = 0 Total nSV = 23