ElasticSearch: Unassigned Shards, how to fix?

Question

I have an ES cluster with 4 nodes:

number_of_replicas: 1
search01 - master: false, data: false
search02 - master: true, data: true
search03 - master: false,          


        

Answer 1

I was having this issue as well, and I found an easy way to resolve it.

• Get the index of unassigned shards

  $ curl -XGET http://172.16.4.140:9200/_cat/shards
  

• Install curator Tools, and use it to delete index

  $ curator --host 172.16.4.140 delete indices --older-than 1 \
         --timestring '%Y.%m.%d' --time-unit days --prefix logstash
  

  NOTE: In my case, the index is logstash of the day 2016-04-21

• Then check the shards again, all the unassigned shards go away!

Answer 2

I just first increased the

"index.number_of_replicas"

by 1 (wait until nodes are synced) then decreased it by 1 afterwards, which effectively removes the unassigned shards and cluster is Green again without the risk of losing any data.

I believe there are better ways but this is easier for me.

Hope this helps.

Answer 3

In my case, when I create a new index then the default number_of_replicas is set as 1. And the number of nodes in my cluster was only one so there was no extra node to create the replica, so the health was turning to yellow. So when I created the index with settings property and set the number_of_replicas as 0. Then it worked fine. Hope this helps.

PUT /customer
{
    "settings": {
        "number_of_replicas": 0
    }
}

Answer 4

I also encountered similar error. It happened to me because one of my data node was full and due to which shards allocation failed. If unassigned shards are there and your cluster is RED and few indices also RED, in that case I have followed below steps and these worked like a champ.
in kibana dev tool-

GET _cluster/allocation/explain

If any unassigned shards are there then you will get details else will throw ERROR.

simply running below command will solve everything-

POST _cluster/reroute?retry_failed

Thanks to -
https://github.com/elastic/elasticsearch/issues/23199#issuecomment-280272888

Answer 5

When dealing with corrupted shards you can set the replication factor to 0 and then set it back to the original value. This should clear up most if not all your corrupted shards and relocate the new replicas in the cluster.

Setting indexes with unassigned replicas to use a replication factor of 0:

curl -XGET http://localhost:9200/_cat/shards |\
  grep UNASSIGNED | grep ' r ' |\
  awk '{print $1}' |\
  xargs -I {} curl -XPUT http://localhost:9200/{}/_settings -H "Content-Type: application/json" \
  -d '{ "index":{ "number_of_replicas": 0}}'

Setting them back to 1:

curl -XGET http://localhost:9200/_cat/shards |\
  awk '{print $1}' |\
  xargs -I {} curl -XPUT http://localhost:9200/{}/_settings -H "Content-Type: application/json" \
  -d '{ "index":{ "number_of_replicas": 1}}'

Note: Do not run this if you have different replication factors for different indexes. This would hardcode the replication factor for all indexes to 1.

Answer 6

I tried to delete unassigned shards or manually assign them to particular data node. It didn't work because unassigned shards kept appearing and health status was "red" over and over. Then I noticed that one of the data nodes stuck in "restart" state. I reduce number of data nodes, killed it. Problem is not reproducible anymore.