High disk watermark exceeded even when there is not much data in my index

Question

I\'m using elasticsearch on my local machine. The data directory is only 37MB in size but when I check logs, I can see:

[2015-05-17 21:31:12,905][WARN

Answer 1

Instead of percentage I use absolute values and rise values for better space use (in pre-prod):

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.threshold_enabled": true,
    "cluster.routing.allocation.disk.watermark.low": "1g",
    "cluster.routing.allocation.disk.watermark.high": "500m",
    "cluster.info.update.interval": "5m" 
  }
}

Also I reduce pooling interval to make ES logs shorter ))

Answer 2

From Index Shard Allocation:

... watermark.high controls the high watermark. It defaults to 90%, meaning ES will attempt to relocate shards to another node if the node disk usage rises above 90%.

The size of your actual index doesn't matter; it's the free space left on the device which matters.

If the defaults are not appropriate for you, you've to change them.

Answer 3

this slightly modified curl command from the Elasticsearch 6.4 docs worked for me:

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "2gb",
    "cluster.routing.allocation.disk.watermark.high": "1gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "500mb",
    "cluster.info.update.interval": "1m"
  }
}
'

if the curl -XPUT command succeeds, you should see logs like this in the Elasticsearch terminal window:

[2018-08-24T07:16:05,584][INFO ][o.e.c.s.ClusterSettings  ] [bhjM1bz] updating [cluster.routing.allocation.disk.watermark.low] from [85%] to [2gb]
[2018-08-24T07:16:05,585][INFO ][o.e.c.s.ClusterSettings  ] [bhjM1bz] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [1gb]
[2018-08-24T07:16:05,585][INFO ][o.e.c.s.ClusterSettings  ] [bhjM1bz] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [500mb]
[2018-08-24T07:16:05,585][INFO ][o.e.c.s.ClusterSettings  ] [bhjM1bz] updating [cluster.info.update.interval] from [30s] to [1m]

https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

Answer 4

Its a warning and won't affect anything. The storage processors (SPs) use high and low watermarks to determine when to flush their write caches. The possible solution can be to free some memory

And the warning will disappear. Even with it showing, the replicas will not be assigned to the node which is okay. The elasticsearch will work fine.

Answer 5

To resolve the issue in which, the log is recorded as:

high disk watermark [90%] exceeded on [ytI5oTyYSsCVfrB6CWFL1g][ytI5oTy][/var/lib/elasticsearch/nodes/0] free: 552.2mb[4.3%], shards will be relocated away from this node

You can update the threshold limit by executing following curl request:

curl -XPUT "http://localhost:9200/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'

Answer 6

Clear up some space on your hard drive. That should fix the issue. This shall also change the health of your ES clusters to Green from Yellow (if you got the above issue, you are most likely to face the yellow cluster health issue as well).