Analyzers in elasticsearch

Question

I'm having trouble understanding the concept of analyzers in elasticsearch with tire gem. I'm actually a newbie to these search concepts. Can someone here help me with some reference article or explain what actually the analyzers do and why they are used?

I see different analyzers being mentioned at elasticsearch like keyword, standard, simple, snowball. Without the knowledge of analyzers I couldn't make out what actually fits my need.

Answer 1

Let me give you a short answer.

An analyzer is used at index Time and at search Time. It's used to create an index of terms.

To index a phrase, it could be useful to break it in words. Here comes the analyzer.

It applies tokenizers and token filters. A tokenizer could be a Whitespace tokenizer. It split a phrase in tokens at each space. A lowercase tokenizer will split a phrase at each non-letter and lowercase all letters.

A token filter is used to filter or convert some tokens. For example, a ASCII folding filter will convert characters like ê, é, è to e.

An analyzer is a mix of all of that.

You should read Analysis guide and look at the right all different options you have.

By default, Elasticsearch applies the standard analyzer. It will remove all common english words (and many other filters)

You can also use the Analyze Api to understand how it works. Very useful.

Answer 2

In Lucene, an analyzer is a combination of tokenizer (splitter) + stemmer + stopword filter

In ElasticSearch, an analyzer is a combination of

1. Character filter: "tidy up" a string before it is tokenized. Example: remove HTML tags
2. Tokenizer: MUST have a single tokenizer. It's used to break up the string into individual terms or tokens
3. Token filter: change, add or remove tokens. Stemmer is an example of token filter, it is used to get base of the word, for example: "happy", "happiness" => "happi".

See Snowball demo here

This is the setting for my system:

     {
      "settings":{
        "index" : {
            "analysis" : {
                "analyzer" : {
                    "analyzerWithSnowball" : {
                        "tokenizer" : "standard",
                        "filter" : ["standard", "lowercase", "englishSnowball"]
                    }
                },
                "filter" : {
                    "englishSnowball" : {
                        "type" : "snowball",
                        "language" : "english"
                    }
                }
            }
        }
      }
    }

Ref:

1. Comparison of Lucene Analyzers
2. http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-analyzers.html

Answer 3

Here's an awesome plugin on github repo. It's somewhat extension of Analyze API. Found it on official elastic plugin list.

What's great is that it shows tokens with all their attributes after every single step. With this it is easy to debug analyzer configuration and see why we got such tokens and where we lost the ones we wanted.

Wish I had found it earlier than today. Thanks to that I just found out why my keyword_repeat token tokenizer seemed to not work correctly. The problem was caused by next token filter: icu_transform (used for transliteration) which unfortunately didn't respect keyword attribute and transformed all of the tokens. Don't know how else would I find the cause if not for this plugin.