Elasticsearch cluster 'master_not_discovered_exception'

Question

i have installed elasticsearch 2.2.3 and configured in cluster of 2 nodes

Node 1 (elasticsearch.yml)

cluster.name: my-cluster
node.name: node1
bootstrap.         


        

Answer 1

There's a lot of settings in here that you either don't want (like the fielddata one) or don't need. Also, you're clearly using AWS EC2 instances, so you should use the cloud-aws plugin (broken into separate plugins in ES 5.x). This will provide a new discovery model that you can take advantage of instead of zen.

For each node, you'll want to therefore install the cloud-aws plugin (assuming ES 2.x):

$ bin/plugin install cloud-aws

Once installed on each node, then you can use it to take advantage of the discovery-ec2 component:

# Guarantee that the plugin is installed
plugin.mandatory: cloud-aws

# Discovery / AWS EC2 Settings
discovery
  type: ec2
  ec2:
    availability_zones: [ "us-east-1a", "us-east-1b" ]
    groups: [ "my_security_group1", "my_security_group2" ]

cloud:
  aws
    access_key: AKVAIQBF2RECL7FJWGJQ
    secret_key: vExyMThREXeRMm/b/LRzEB8jWwvzQeXgjqMX+6br
    region: us-east-1
  node.auto_attributes: true

# Bind to the network on whatever IP you want to allow connections on.
# You _should_ only want to allow connections from within the network
# so you only need to bind to the private IP
node.host: _ec2:privateIp_

# You can bind to all hosts that are possible to communicate with the
# node but advertise it to other nodes via the private IP (less
# relevant because of the type of discovery used, but not a bad idea).
#node:
#  bind_host: [ _ec2:privateIp_, _ec2:publicIp_, _ec2:publicDns_ ]
#  publish_host: _ec2:privateIp_

# Node-specific settings (note: nodes default to be master and data nodes)
node:
  name: node1
  master: true
  data: true

# Constant settings
cluster.name: my-cluster
bootstrap.mlockall: true

Finally, your problem is that you are failing master election for some reason that most likely stems from connectivity issues. The above configuration should fix those issues, but you have one other critical issue: you are specifying the discovery.zen.minimum_master_nodes setting incorrectly. You have two eligible master nodes, but you are asking Elasticsearch to require only one for any election. That means, in isolation, each eligible master node can decide that they have a quorum, and therefore elect themselves separately (thus giving two masters and effectively two clusters). This is bad.

You must therefore always set that setting using quorum: (M / 2) + 1, rounded down, where M is the number of master eligible nodes. So:

M = 2
(2 / 2) + 1 = (1) + 1 = 2

If you had 3, 4, or 5 master eligible nodes, then it would be:

M = 3
(3 / 2) + 1 = (1.5) + 1 = 2.5 => 2

M = 4
(4 / 2) + 1 = (2) + 1 = 3

M = 5
(5 / 2) + 1 = (2.5) + 1 = 3.5 => 3

So, you should also be setting, in your case:

discovery.zen.minimum_master_nodes: 2

Note, you could add this either as another line or, you could modify the discovery block from above (it really comes down to style of YAML):

discovery
  type: ec2
  ec2:
    availability_zones: [ "us-east-1a", "us-east-1b" ]
    groups: [ "my_security_group1", "my_security_group2" ]
  zen.minimum_master_nodes: 2