ElasticSearch: Unassigned Shards, how to fix?

Question

I have an ES cluster with 4 nodes:

number_of_replicas: 1
search01 - master: false, data: false
search02 - master: true, data: true
search03 - master: false,          


        

Answer 1

This may be a cause of the disk space as well, In Elasticsearch 7.5.2, by default, if disk usage is above 85% then replica shards are not assigned to any other node.

This can be fixed by setting a different threshold or by disabling it either in the .yml or via Kibana

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.threshold_enabled": "false"
  }
}

Answer 2

This little bash script will brute force reassign, you may lose data.

NODE="YOUR NODE NAME"
IFS=$'\n'
for line in $(curl -s 'localhost:9200/_cat/shards' | fgrep UNASSIGNED); do
  INDEX=$(echo $line | (awk '{print $1}'))
  SHARD=$(echo $line | (awk '{print $2}'))

  curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
     "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
          }
        }
    ]
  }'
done

Answer 3

Maybe it helps someone, but I had the same issue and it was due to a lack of storage space caused by a log getting way too big.

Hope it helps someone! :)

Answer 4

In my case an old node with old shares was joining the cluster, so we had to shutdown the old node and delete the indices with unassigned shards.

Answer 5

Elasticsearch automatically allocates shards if the below config is set to all. This config can be set using a rest api as well cluster.routing.allocation.enable: all

If even after application of the below config, es fails to assign the shards automatically, then you have to force assign the shards yourself. ES official link for this

I have written a script to force assign all unassigned shards across cluster.

below array contains list of nodes among which you want to balance the unassigned shards

#!/bin/bash
array=( node1 node2 node3 )
node_counter=0
length=${#array[@]}
IFS=$'\n'
for line in $(curl -s 'http://127.0.0.1:9200/_cat/shards'|  fgrep UNASSIGNED); do
    INDEX=$(echo $line | (awk '{print $1}'))
    SHARD=$(echo $line | (awk '{print $2}'))
    NODE=${array[$node_counter]}
    echo $NODE
    curl -XPOST 'http://127.0.0.1:9200/_cluster/reroute' -d '{
        "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
            }
        }
        ]
    }'
    node_counter=$(((node_counter)%length +1))
done

Answer 6

I also meet this situation and finally fixed it.

Firstly, I will describe my situation. I have two nodes in ElasticSearch cluster, they can find each other, but when I created a index with settings "number_of_replicas" : 2, "number_of_shards" : 5, ES show yellow signal and unassigned_shards is 5.

The problem occurs because the value of number_of_replicas, when I set its value with 1, all is fine.