Increasing the size of the queue in Elasticsearch?

Question

Ive been looking at my elasticsearch logs, and I came across the error

rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServ

Answer 1

To change the queue size one could add it in the config file for each of the nodes as follows:

threadpool.search.queue_size: <new queue size> .

However this would also require a cluster restart.

Up to Elasticsearch 2.x, you can update via the cluster-setting api and this would not require a cluster restart, however this option is gone with Elasticsearch 5.x and newer.

curl -XPUT  _cluster/settings -d '{
    "persistent" : {
        "threadpool.search.queue_size" : <new_size>
    }
}'

You can query the queue size as follows:

curl <server>/_cat/thread_pool?v&h=search.queueSize

Answer 2

As of Elasticsearch 6 the type of the search thread pool has changed to fixed_auto_queue_size, which means setting threadpool.search.queue_size in elasticsearch.yml is not enough, you have to control the min_queue_size and max_queue_size parameters as well, like this:

thread_pool.search.queue_size: <new_size>
thread_pool.search.min_queue_size: <new_size>
thread_pool.search.max_queue_size: <new_size>

I recommend using _cluster/settings?include_defaults=true to view the current thread pool settings in your nodes before making any changes. For more information about the fixed_auto_queue_size thread pool read the docs.

Answer 3

From Elasticsearch 5 onward you cannot use the API to update the threadpool search queue size. It is now a node-level settings. See this.

Thread pool settings are now node-level settings. As such, it is not possible to update thread pool settings via the cluster settings API.

To update the threadpool you have to add thread_pool.search.queue_size : <New size> in elasticsearch.yml file of each node and then restart elasticsearch.