Spark with Webhdfs/httpfs

问题

I would like to read a file from HDFS into Spark via httpfs or Webhdfs. Something along the lines of

sc.textFile("webhdfs://myhost:14000/webhdfs/v1/path/to/file.txt")

or, ideally,

sc.textFile("httpfs://myhost:14000/webhdfs/v1/path/to/file.txt")

Is there a way to get Spark to read the file over Webhdfs/httpfs?

回答1:

I believe WebHDFS/ HttpFS are like streaming sources to transmit the data over REST-API.

Then Spark Streaming can be used to receive the data from the WebHDFS/ HttpFS.

回答2:

According to SPARK-2930 document enhancement request, spark.yarn.access.namenodes should also works for webhdfs / hdfs. SPARK-2930 clarify docs on using webhdfs with spark.yarn.access.namenodes

Running Spark on YARN Get more details about spark.yarn.access.namenodes

来源：https://stackoverflow.com/questions/27367962/spark-with-webhdfs-httpfs