Namenode HA (UnknownHostException: nameservice1)

Question

We enable Namenode High Availability through Cloudera Manager, using

Cloudera Manager >> HDFS >> Action > Enable High Availability >> Selected Stand By Namenode &a

Answer 1

Got it resolved. wrong config was linked to "/etc/hadoop/conf/" --> "/etc/alternatives/hadoop-conf/" --> "/etc/hadoop/conf.cloudera.mapreduce1"

It has to be "/etc/hadoop/conf/" --> "/etc/alternatives/hadoop-conf/" --> "/etc/hadoop/conf.cloudera.mapreduce"

Answer 2

below statement in my code resolved problem by specifying the host and port

val dfs = sqlContext.read.json("hdfs://localhost:9000//user/arvindd/input/employee.json")

Answer 3

Looks like you are using wrong client configuration in /etc/hadoop/conf directory. Sometimes Cloudera Manager (CM) deploy client configurations option may not work.

As you have enabled NN HA, you should have valid core-site.xml and hdfs-site.xml files in your hadoop client configuration directory. For getting the valid site files, Go to HDFS service from CM Choose Download client configuration option from the Actions Button. you will get configuration files in zip format, extract the zip files and replace /etc/hadoop/conf/core-site.xml and /etc/hadoop/conf/hdfs-site.xml files with the extracted core-site.xml,hdfs-site.xml files.

Answer 4

I resolved this issue my putting the complete line to create RDD

myfirstrdd = sc.textFile("hdfs://192.168.35.132:8020/BUPA.txt")

and then I was able to do other RDD transformation .. Make sure you have the w/r/x to the file or you can do chmod 777