问题
When tokenizing a string of text, I need to extract the indexes of the tokenized words. For example, given:
"Mary didn't kiss John"
I would need something like:
[(Mary, 0), (did, 5), (n't, 8), (kiss, 12), (John, 17)]
Where 0, 5, 8, 12 and 17 correspond to the index (in the original string) where the token began. I cannot rely on just whitespace, since some words become 2 tokens. Further, I cannot just search for the token in the string, since the word likely will appear multiple times.
One giant obstacle is that I'm working with "dirty" text. Here is a real example from the corpus, and its tokenization:
String:
The child some how builds a boaty c capable of getting scrtoacross the sea, even after findingovercoming many treachrous rous obsittalcles.
Tokens:
The, child, some, how, builds, a, boaty, , , c, , capable, of, getting, scrto, , across, the, sea, ,, even, after, finding, , , , , overcoming, many, treachrous, rous, obsittalcles, .
I'm currently using OpenNLP to tokenize the text, but am ambivalent about which API to utilize for tokenization. It does need to be Java, though, so (unfortunately) Python's NLTK is out of the picture.
Any ideas would be greatly appreciated! Thanks!
回答1:
OpenNLP will return the offsets using the method Tokenizer.tokenizePos(String s), see the OpenNLP API for TokenizerME as an example for one the implemented tokenizers. Each Span returned contains the start and end positions of the token.
Whether you decide to use UIMA is really a separate question, but OpenNLP does provide UIMA annotators for their tokenizers that use tokenizePos(). However, if you just want to tokenize a string, UIMA is definitely overkill...
回答2:
You can use OpenNLP Tokenizer with UIMA. The token annotator in UIMA will create a type for Token which will include the start and end indices of the token. You can also attach features like part-of-speech tag, stem, lemma, etc. to the token. UIMA has Java and C++ APIs.
回答3:
The same you can do with BreakIterator instead of using any external API.
来源：https://stackoverflow.com/questions/12398706/java-nlp-extracting-indicies-when-tokenizing-text