Elasticsearch 2.1: Result window is too large (index.max_result_window)

Question

We retrieve information from Elasticsearch 2.1 and allow the user to page thru the results. When the user requests a high page number we get the following error message:

Answer 1

2) It does not seem that the scrolling API is an option for me but that I have to increase "index.max_result_window". Does anyone have any experience with this?

--> You can define this value in index templates , es template will be applicable for new indexes only ,so you either have to delete old indexes after creating template or wait for new data to be ingested in elasticsearch .

{ "order": 1, "template": "index_template*", "settings": { "index.number_of_replicas": "0", "index.number_of_shards": "1", "index.max_result_window": 2147483647 },

Answer 2

The right solution would be to use scrolling.
However, if you want to extend the results search returns beyond 10,000 results, you can do it easily with Kibana:

Go to Dev Tools and just post the following to your index (your_index_name), specifing what would be the new max result window

PUT your_index_name/_settings
{ 
  "max_result_window" : 500000 
}

If all goes well, you should see the following success response:

{
  "acknowledged": true
}

Answer 3

In my case it looks like reducing the results via the from & size prefixes to the query will remove the error as we don't need all the results:

GET widgets_development/_search
{
  "from" : 0, 
  "size": 5,
  "query": {
    "bool": {}
  },
  "sort": {
    "col_one": "asc"
  }
}

Answer 4

The following pages in the elastic documentation talk about deep paging:

https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html https://www.elastic.co/guide/en/elasticsearch/guide/current/_fetch_phase.html

Depending on the size of your documents, the number of shards, and the hardware you are using, paging 10,000 to 50,000 results (1,000 to 5,000 pages) deep should be perfectly doable. But with big-enough from values, the sorting process can become very heavy indeed, using vast amounts of CPU, memory, and bandwidth. For this reason, we strongly advise against deep paging.

Answer 5

If you need deep pagination, one possible solution is to increase the value max_result_window. You can use curl to do this from your shell command line:

curl -XPUT "http://localhost:9200/my_index/_settings" -H 'Content-Type: application/json' -d '{ "index" : { "max_result_window" : 500000 } }'

I did not notice increased memory usage, for values of ~ 100k.

Answer 6

If you want more than 10000 results then in all the data nodes the memory usage will be very high because it has to return more results in each query request. Then if you have more data and more shards then merging those results will be inefficient. Also es cache the filter context, hence again more memory. You have to trial and error how much exactly you are taking. If you are getting many requests in small window you should do multiple query for more than 10k and merge it by urself in the code, which is supposed to take less application memory then if you increase the window size.