Spark - failed on connection exception: java.net.ConnectException - localhost

Question

I am running hadoop and spark in a single machine (Ubuntu 14.04). JPS command gives me below output

hduser@ubuntu:~$ jps
4370 HRegionServer
6568 Jps
5555 Run         


        

Answer 1

Sorry... actually namenode is running in a different port which I found from core-site.xml. It works for me after using the port 54310.

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>

Could someone please tell me what does 3079 means here, which is the output of JPS command.

Answer 2

That's a good answer to yourself ! :)

3079 is the pid ( process id ) of namenode. See more details about jps's output here: Jps doc. BTW, you can verify the listening ports of namenode using the pid with a command such as:

lsof -p 3079 -a -i

In your command output:

java    3079 hduser   89u  IPv4  59998      0t0  TCP localhost:54310->localhost:46507 (ESTABLISHED)

That line indicates that other process established TCP connection with Namenode at port 46507. Possible processes could be resoure manager(RM) or data nodes etc.