What exactly does the Standard tokenfilter do in Elasticsearch?

Question

There are no examples in the documentation, and I was just wondering what to expect from input I give it.

Answer 1

Before Elasticsearch 0.16 (Lucene 3.1) the standard token filter was "normalizing tokens extracted by standard tokenizer". To be specific, it was removing 's at the end of words and dots in acronyms. So, back then Apple's C.E.O would become Apple CEO after passing the standard filter. Starting with Elasticsearch 0.16 (Lucene 3.1) the standard token filter does nothing (at least at the moment). It simply passes tokens to the next filter in the chain.

Answer 2

Here's an excerpt of the source code of StandardFilter from Lucene trunk. It really does nothing.

public final boolean incrementToken() throws IOException {
    return input.incrementToken(); // TODO: add some niceties for the new grammar
}