stemming

问题 I am running the following code on IDLE(Python) and I want to enter Arabic string and get the stemming for it but actually it doesn't work ">>> from nltk.stem.isri import ISRIStemmer ">>> st = ISRIStemmer() ">>> w= 'حركات' ">>> join = w.decode('Windows-1256') ">>> print st.stem(join).encode('Windows-1256').decode('utf-8') The result of running it is the same text in w which is 'حركات' which is not the stem but when do the following: ">>> print st.stem(u'اعلاميون') the result succeeded and

问题 Stemming is something that's needed in tagging systems. I use delicious, and I don't have time to manage and prune my tags. I'm a bit more careful with my blog, but it isn't perfect. I write software for embedded systems that would be much more functional (helpful to the user) if they included stemming. For instance: Parse Parser Parsing Should all mean the same thing to whatever system I'm putting them into. Ideally there's a BSD licensed stemmer somewhere, but if not, where do I look to

问题 I have a file 'check_text.txt' that contains " said say says make made ". I'd like to perform stemming on it to get "say say say make make". I tried to use stemDocument in tm package, as the following, but only get "said say say make made". Is there a way to perform stemming on past tense words? Is it necessary to do so in real-world natural language processing? Thanks! filename = 'check_text.txt' con <- file(filename, "rb") text_data <- readLines(con,skipNul = TRUE) close(con) text_VS <-

问题 I am looking for a class or method that takes a long string of many 100s of words and tokenizes, removes the stop words and stems for use in an IR system. For example: "The big fat cat, said 'your funniest guy i know' to the kangaroo..." the tokenizer would remove the punctuation and return an ArrayList of words the stop word remover would remove words like "the", "to", etc the stemmer would reduce each word the their 'root', for example 'funniest' would become funny Many thanks in advance.

问题 I'm looking for a Java library to extract keywords from a block of text. The process should be as follows: stop word cleaning -> stemming -> searching for keywords based on English linguistics statistical information - meaning if a word appears more times in the text than in the English language in terms of probability than it's a keyword candidate. Is there a library that performs this task? 回答1: Here is a possible solution using Apache Lucene. I didn't use the last version but the 3.6.2 one

问题 I've tried PorterStemmer and Snowball but both don't work on all words, missing some very common ones. My test words are: " cats running ran cactus cactuses cacti community communities ", and both get less than half right. See also: Stemming algorithm that produces real words Stemming - code examples or open source projects? 回答1: If you know Python, The Natural Language Toolkit (NLTK) has a very powerful lemmatizer that makes use of WordNet. Note that if you are using this lemmatizer for the

问题 Natural Language Processing (NLP), especially for English, has evolved into the stage where stemming would become an archaic technology if "perfect" lemmatizers exist. It's because stemmers change the surface form of a word/token into some meaningless stems. Then again the definition of the "perfect" lemmatizer is questionable because different NLP task would have required different level of lemmatization. E.g. Convert words between verb/noun/adjective forms. Stemmers [in]: having [out]: hav

问题 I have an Sql Server 2016 database with full text indexes defined on 4 columns, each configured for a different language: Dutch, English, German & French. I used the wizard to setup the full-text index. I am using CONTAINSTABLE with FORMSOF and for each language I would expect executing a query with either the word stem or any verb form would return both results from the example table. This seems to work in English & German, somewhat in French, and not at all in Dutch. I am using a very basic

问题 I am following this document clustering tutorial. As an input I give a txt file which can be downloaded here. It's a combined file of 3 other txt files divided with a use of \n. After creating a tf-idf matrix I received this warning: ,,UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'afterward', 'alon', 'alreadi', 'alway', 'ani', 'anoth', 'anyon', 'anyth', 'anywher', 'becam', 'becaus', 'becom', 'befor', 'besid',

问题 I want to find word stems with Wordnet . Does wordnet have a function for stemming? I use this import for my stemming, but it doesn't work as expected. from nltk.stem.wordnet import WordNetLemmatizer WordNetLemmatizer().lemmatize('Having','v') 回答1: Try using one of the stemmers in nltk.stem module, such as the PorterStemmer. Here's an online demo of NLTK's stemmers: http://text-processing.com/demo/stem/ 回答2: Seems like you have to input a lowercase string to the lemmatize method: >>>