hdfs data node disconnected from namenode

Question

I get from time to time the following errors in cloudera manager:

This DataNode is not connected to one or more of its NameNode(s). 

and

Answer 1

Hadoop uses specific ports to communicate between the DataNode and the NameNode. It could be that a firewall is blocking those specific ports. Check the default ports in the Cloudera WebSite and test the connectivity to the NameNode with specific ports.

Answer 2

I realize you took some steps to test this, but intermittent disconnects still make it sound like a Connectivity issue.

If nodes really don't come back after a disconnect, that may be a configuration issue, which could well be completely independent from the reason why they disconnect in the first place.

Answer 3

If you're using Linux then please make sure that you have configured these properties correctly:

1. Disable SELINUX

type the command getenforce on CLI and if it shows enforcing, means it is enabled. Change it fro /etc/selinux/config file.

2. Disable Firewall

3. Make sure you have NTP service installed.

4. Make sure your server can SSH to all client nodes.

5. Make sure all the nodes have FQDN(Fully Qualified Domain Name) and have an entry in /etc/hosts with name and IP.

If these settings are right in the place then please attach the log of any of your datanode which got disconnected.

Answer 4

I ran into this error

"This DataNode is not connected to one or more of its NameNode(s). "

and I solved it by turning off safe mode and restart HDFS service