tokenize

I am using nltk's PunkSentenceTokenizer to tokenize a text to a set of sentences. However, the tokenizer doesn't seem to consider new paragraph or new lines as a new sentence. >>> from nltk.tokenize.punkt import PunktSentenceTokenizer >>> tokenizer = PunktSentenceTokenizer() >>> tokenizer.tokenize('Sentence 1 \n Sentence 2. Sentence 3.') ['Sentence 1 \n Sentence 2.', 'Sentence 3.'] >>> tokenizer.span_tokenize('Sentence 1 \n Sentence 2. Sentence 3.') [(0, 24), (25, 36)] I would like it to to consider new lines as a boundary of sentences as well. Anyway to do this (I need to save the offsets too

Is there any available solution for (re-)generating PHP code from the Parser Tokens returned by token_get_all ? Other solutions for generating PHP code are welcome as well, preferably with the associated lexer/parser (if any). If I'm not mistaken http://pear.php.net/package/PHP_Beautifier uses token_get_all() and then rewrites the stream. It uses heaps of methods like t_else and t_close_brace to output each token. Maybe you can hijack this for simplicity. wen From my comment: Does anyone see a potential problem, if I simply write a large switch statement to convert tokens back to their string

I have a method that takes in a String parameter, and uses NLTK to break the String down to sentences, then into words. Afterwards, it converts each word into lowercase, and finally creates a dictionary of the frequency of each word. import nltk from collections import Counter def freq(string): f = Counter() sentence_list = nltk.tokenize.sent_tokenize(string) for sentence in sentence_list: words = nltk.word_tokenize(sentence) words = [word.lower() for word in words] for word in words: f[word] += 1 return f I'm supposed to optimize the above code further to result in faster preprocessing time,

I am trying to tokenize and remove stop words from a txt file with Lucene. I have this: public String removeStopWords(String string) throws IOException { Set<String> stopWords = new HashSet<String>(); stopWords.add("a"); stopWords.add("an"); stopWords.add("I"); stopWords.add("the"); TokenStream tokenStream = new StandardTokenizer(Version.LUCENE_43, new StringReader(string)); tokenStream = new StopFilter(Version.LUCENE_43, tokenStream, stopWords); StringBuilder sb = new StringBuilder(); CharTermAttribute token = tokenStream.getAttribute(CharTermAttribute.class); while (tokenStream

I have a cpp file with a huge class implementation. Now I have to modify the source file itself. For this, is there a library/api/tool that will tokenize this file for me and give me one token each time i request. My requirement is as below. OpenCPPFile() While (!EOF) token = GetNextToken(); process something based on this token EndWhile I am happy now Regards, AJ Boost.Wave offers a standard C++ lexer among many other tools like a standard preprocessor which are built on top of Boost.Spirit . Check the following sample in the boost directory: C:\boost\libs\wave\samples\lexed_tokens For

I want to include hyphen words for example: long-term, self-esteem, etc. as a single token in Spacy. After looking at some similar posts on Stackoverflow, Github , its documentation and elsewhere , I also wrote a custom tokenizer as below. import re from spacy.tokenizer import Tokenizer prefix_re = re.compile(r'''^[\[\("']''') suffix_re = re.compile(r'''[\]\)"']$''') infix_re = re.compile(r'''[.\,\?\:\;\...\‘\’\`\“\”\"\'~]''') def custom_tokenizer(nlp): return Tokenizer(nlp.vocab, prefix_search=prefix_re.search, suffix_search=suffix_re.search, infix_finditer=infix_re.finditer, token_match=None

My current web-app project calls for a little NLP: Tokenizing text into sentences, via Punkt and similar; Breaking down the longer sentences by subordinate clause (often it’s on commas except when it’s not) A Bayesian model fit for chunking paragraphs with an even feel, no orphans or widows and minimal awkward splits (maybe) ... which much of that is a childishly easy task if you’ve got NLTK — which I do, sort of: the app backend is Django on Tornado; you’d think doing these things would be a non-issue. However, I’ve got to interactively provide the user feedback for which the tokenizers are