ElasticSearch entered “read only” mode, node cannot be altered

问题

Something happened during the night to my ES cluster (composed of 5 data nodes, 3 master nodes).

To be honest, I have no idea what happened but all the indices and data were deleted and the cluster entered a "read only" mode, possibly hacked?

When trying to get Kiban running i get the following:

Tried restarting kibana - it restarted, nothing changed. Tried restarting Elastic - it restarted (all nodes), nothing changed.

I then had a look at the cluster settings and this is what I got:

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      },
      "blocks": {
        "read_only": "true"
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      }
    }
  }
}

I tried undoing the read only as follows:

PUT _cluster/settings
{
  "persistent": {
    "blocks.read_only": false
  }
}

No luck as you can see:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status": 403
}

Any ideas?

UPDATE: Problem solved by Andrei Stefan, now for the more important part - why? What happened and why? I've lost all data and my cluster entered a read-only mode.

回答1:

The correct command is:

PUT /_cluster/settings
{
  "persistent" : {
    "cluster.blocks.read_only" : false
  }
}

回答2:

It turns out ES has some thresholds for available disk space, and when the "flood" one is hit, it puts the indeces into read only mode.

In order to set it back (tested with ES6), you will need to do the following:

PUT /[index_name]/_settings
{
  "index.blocks.read_only_allow_delete": null
}

More information can be found on the following page of the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

来源：https://stackoverflow.com/questions/38735514/elasticsearch-entered-read-only-mode-node-cannot-be-altered