tokenize

问题 I have a text classification problem where i have two types of features: features which are n-grams (extracted by CountVectorizer) other textual features (e.g. presence of a word from a given lexicon). These features are different from n-grams since they should be a part of any n-gram extracted from the text. Both types of features are extracted from the text's tokens. I want to run tokenization only once,and then pass these tokens to CountVectorizer and to the other presence features

问题 I am new to Spacy and NLP. Facing the below issue while doing sentence segmentation using Spacy. The text I am trying to tokenise into sentences contains numbered lists(with space between numbering and actual text) . Like below. import spacy nlp = spacy.load('en_core_web_sm') text = "This is first sentence.\nNext is numbered list.\n1. Hello World!\n2. Hello World2!\n3. Hello World!" text_sentences = nlp(text) for sentence in text_sentences.sents: print(sentence.text) Output (1.,2.,3. are

问题 I have a cpp file with a huge class implementation. Now I have to modify the source file itself. For this, is there a library/api/tool that will tokenize this file for me and give me one token each time i request. My requirement is as below. OpenCPPFile() While (!EOF) token = GetNextToken(); process something based on this token EndWhile I am happy now Regards, AJ 回答1: Boost.Wave offers a standard C++ lexer among many other tools like a standard preprocessor which are built on top of Boost

问题 What are practical and day-to-day usage examples of PHP Tokenizer ? Has anyone used this? 回答1: I use PHP_CodeSniffer for coding style compliance, which is built on the tokeniser. Also, some frameworks (e.g. Symfony 2) use the tokeniser to generate cache files or intermediate class files of PHP code. It's also possible to use the tokeniser to build a source code formatter or syntax highlighter. Basically, anywhere you use PHP code as data you can use the tokeniser. It's much more reliable that

问题 I am trying tokenize strings into ngrams. Strangely in the documentation for the NGramTokenizer I do not see a method that will return the individual ngrams that were tokenized. In fact I only see two methods in the NGramTokenizer class that return String Objects. Here is the code that I have: Reader reader = new StringReader("This is a test string"); NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3); Where are the ngrams that were tokenized? How can I get the output in Strings

问题 I need to convert a list of numbers that fall within certain ranges into a list of values, ordered by a priority column. The table has the following values: | YEAR | R_MIN | R_MAX | VAL | PRIO | ------------------------------------ 2010 18000 90100 52 6 2010 240000 240099 82 3 2010 250000 259999 50 5 2010 260000 260010 92 1 2010 330000 330010 73 4 2010 330011 370020 50 5 2010 380000 380050 84 2 The ranges will be different for different years. The ranges within one year will never overlap.

问题 I'm beginner of lucene. Here's my source: ft = new FieldType(StringField.TYPE_STORED); ft.setTokenized(false); ft.setStored(true); ftNA = new FieldType(StringField.TYPE_STORED); ftNA.setTokenized(true); ftNA.setStored(true); Why tokenized in lucene? For example: the String value of "my name is lee" case tokenized, "my" "name" "is" "lee" case not tokenized, "my name is lee" I'dont understand why indexing by tokenized. What is the difference between tokenized and not tokenized? 回答1: Lucene

问题 I'd like to write a custom tokenizer in Apache Solr 3.6, that will try to identify certain types of tokens and depending on their type search on different fields. For example, if the search were for 2in screw , I would have the custom tokenizer identify that 2in refers to a value and unit of measure and that screw refers to a category. I would then want to limit the search to "in" in the measure field and "screw" in the category field. I realize that this can be done be externally breaking

问题 i want to search a specific word(which is entered by user) in .xml file. This is my xml file. <?xml version="1.0" encoding="UTF-8"?> <words> <entry> <word>John</word> <pron>()</pron> <gram>[Noun]</gram> <poem></poem> <meanings> <meaning>name</meaning> </meanings> </entry> </words> here is my Code import nltk from nltk.tokenize import word_tokenize import os import xml.etree.ElementTree as etree sen = input("Enter Your sentence - ") print(sen) print("\n") print(word_tokenize(sen)[0]) tree =

问题 I have a string, which looks like this: foo $RESULT :(0.2374742, 0.267722, ...up to a million more) $STATES :{1, 3, 5, ...} foo so somewhere in the string are results and directly after them are the states and I want to save the Results in a list and the states in another list. I think I need something like "read from $RESULT :(" to ")" get every number and push to list, same for States, but I dont know how to read a String from "a" to "b" and tokenize its content. 回答1: you could use boost