Zookeeper connection error

Question

We have a standalone zookeeper setup on a dev machine. It works fine for every other dev machine except this one testdev machine.

We get this error over and over aga

Answer 1

Check the zookeeper logs (/var/log/zookeeper). It looks like a connection is established, which should mean there is a record of it.

I had the same situation and it was because a process opened connections and failed to close them. This eventually exceeded the per-host connection limit and my logs were overflowing with

2016-08-03 15:21:13,201 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@188] - Too many connections from /172.31.38.64 - max is 50

Assuming zookeeper is on the usual port, you could do a check for that with:

lsof -i -P | grep 2181

Answer 2

I was having the same error when I was trying to connect my broker with my Zookeeper ensemble using A records to point to Zookeeper IPs. The problem was in my zookeepers. My zookeepers were not able to bind to port 2181 because I was pointing my A records to public IP. This was preventing the zookeeper ensemble to choose a leader and communicate with each other. Pointing A records to private IP enabled the zookeeper ensemble to choose a leader and the cluster became active. After this when I tried to connect one of my brokers to the ensemble, it connected successfully.

Answer 3

I had this problem too, and I found that I just need to restart zookeeper, then restart tomcat so my webapp connected nicely then

Answer 4

I encountered same problem ,too. In my case the problem is about iptables rules.

To communicate with zookeeper node, 2181 port must be accept incoming request, also for internal communication between zookeeper nodes 2888,3888 ports must be open for incoming request.

iptables -t nat -I PREROUTING -p tcp -s 10.0.0.0/24 --dport 2181 -j DNAT --to-destination serverIp:2181
iptables -t nat -I PREROUTING -p udp -s 10.0.0.0/24 --dport 2181 -j DNAT --to-destination serverIp:2181

iptables -t nat -I PREROUTING -p tcp -s 10.0.0.0/24 --dport 2888 -j DNAT --to-destination serverIp:2888
iptables -t nat -I PREROUTING -p udp -s 10.0.0.0/24 --dport 2888 -j DNAT --to-destination serverIp:2888

iptables -t nat -I PREROUTING -p tcp -s 10.0.0.0/24 --dport 3888 -j DNAT --to-destination serverIp:3888
iptables -t nat -I PREROUTING -p udp -s 10.0.0.0/24 --dport 3888 -j DNAT --to-destination serverIp:3888

sudo service iptables save

Answer 5

I had this problem too, and it turned out that I was telling zookeeper to connect to the wrong port. Have you verified that zookeeper is actually running on port 2181 on the dev machine?

Answer 6

I just have the same situation as you and I have just fixed this problem.

It is the reason that you have configured even number of zookeepers which directly result in this problem,try to change your number of zookeeper node to a odd one.

for example the original status of my zookeeper cluster is consisted of 4 nodes,then just remove one of them which result in the number of node to be 3 well ,now its ok to startup zookeeper cluster

below is the output of successfully connect to zookeeper server

2013-04-22 22:07:05,654 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1321ed6
Welcome to ZooKeeper!
2013-04-22 22:07:05,704 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@966] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2013-04-22 22:07:05,727 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@849] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2013-04-22 22:07:05,846 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13e3211c06e0000, negotiated timeout = 30000