Hadoop cluster setup - java.net.ConnectException: Connection refused

Question

I want to setup a hadoop-cluster in pseudo-distributed mode. I managed to perform all the setup-steps, including startuping a Namenode, Datanode, Jobtracker and a Tasktracke

Answer 1

I am also facing same issue in Hortonworks

At the time I restart the Ambari agents and servers then the issue has been resolved.

 systemctl stop ambari-agent 

systemctl stop ambari-server

Source :Full Article With Resolution

 systemctl start ambari-agent

systemctl start ambari-server

Answer 2

For me it was that I could not cluster my zookeeper.

hdfs haadmin -getServiceState 1
active

hdfs haadmin -getServiceState 2
active

My hadoop-hdfs-zkfc-[hostname].log showed:

2017-04-14 11:46:55,351 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at HOST/192.168.1.55:9000: java.net.ConnectException: Connection refused Call From HOST/192.168.1.55 to HOST:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

solution:

hdfs-site.xml
  <property>
    <name>dfs.namenode.rpc-bind-host</name>
      <value>0.0.0.0</value>
  </property>

before

netstat -plunt

tcp        0      0 192.168.1.55:9000        0.0.0.0:*               LISTEN      13133/java

nmap localhost -p 9000

Starting Nmap 6.40 ( http://nmap.org ) at 2017-04-14 12:15 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000047s latency).
Other addresses for localhost (not scanned): 127.0.0.1
PORT     STATE  SERVICE
9000/tcp closed cslistener

after

netstat -plunt
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      14372/java

nmap localhost -p 9000

Starting Nmap 6.40 ( http://nmap.org ) at 2017-04-14 12:28 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000039s latency).
Other addresses for localhost (not scanned): 127.0.0.1
PORT     STATE SERVICE
9000/tcp open  cslistener

Answer 3

Your issue is a very interesting one. Hadoop setup could be frustrating some time due to the complexity of the system and many moving parts involved. I think the issue you faced is definitely a firewall one. My hadoop cluster has similar setup. With a firewall rule added with command:

 sudo iptables -A INPUT -p tcp --dport 9000 -j REJECT

I'm able to see the exact issue:

15/03/02 23:46:10 INFO client.RMProxy: Connecting to ResourceManager at  /0.0.0.0:8032
java.net.ConnectException: Call From mybox/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

You can verify your firewall settings with command:

/usr/local/hadoop/etc$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             anywhere             tcp dpt:9000 reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination   

Once the suspicious rule is identified, it could be deleted with a command like:

 sudo iptables -D INPUT -p tcp --dport 9000 -j REJECT 

Now, the connection should go through.