webhdfs

问题 I am trying to open a hdfs file that is present on cdh4 cluster from cdh5 machine using webhdfs from the command line as below: curl -i -L "http://namenodeIpofCDH4:50070/webhdfs/v1/user/quad/source/JSONML.java?user.name=quad&op=OPEN" I am getting "File Not Found Exception" even if the file JSONML.java is present in the mentioned path in namenode as well as datanode and its trace is as follows: HTTP/1.1 307 TEMPORARY_REDIRECT Cache-Control: no-cache Expires: Thu, 01-Jan-1970 00:00:00 GMT Date:

问题 I am trying to open a hdfs file that is present on cdh4 cluster from cdh5 machine using webhdfs from the command line as below: curl -i -L "http://namenodeIpofCDH4:50070/webhdfs/v1/user/quad/source/JSONML.java?user.name=quad&op=OPEN" I am getting "File Not Found Exception" even if the file JSONML.java is present in the mentioned path in namenode as well as datanode and its trace is as follows: HTTP/1.1 307 TEMPORARY_REDIRECT Cache-Control: no-cache Expires: Thu, 01-Jan-1970 00:00:00 GMT Date:

问题 Is there a way or any command using which I can come to know the disk space of each datanode or the total cluster disk space? I tried the command dfs -du -h / but it seems that I do not have permission to execute it for many directories and hence cannot get the actual disk space. 回答1: From UI: http://namenode:50070/dfshealth.html#tab-datanode ---> which will give you all the details about datanode. From command line: To get disk space of each datanode: sudo -u hdfs hdfs dfsadmin -report --->

问题 I would like to read a file from HDFS into Spark via httpfs or Webhdfs. Something along the lines of sc.textFile("webhdfs://myhost:14000/webhdfs/v1/path/to/file.txt") or, ideally, sc.textFile("httpfs://myhost:14000/webhdfs/v1/path/to/file.txt") Is there a way to get Spark to read the file over Webhdfs/httpfs? 回答1: I believe WebHDFS/ HttpFS are like streaming sources to transmit the data over REST-API. Then Spark Streaming can be used to receive the data from the WebHDFS/ HttpFS. 回答2:

问题 I'm working on using the REST interface to Hadoop's HDFS as a convenient way to store files over the network. To test I installed hadoop on my mac (10.8.5) following these instructions: http://importantfish.com/how-to-install-hadoop-on-mac-os-x/ That worked like a charm and I'm able to start hadoop and run a basic test: hadoop-examples-1.1.2.jar pi 10 100 Now, I'm using the python client to handle the HTTP requests to/from webhdfs: http://pythonhosted.org/pywebhdfs/ But I'm stumbling on a

问题 I'm working on using the REST interface to Hadoop's HDFS as a convenient way to store files over the network. To test I installed hadoop on my mac (10.8.5) following these instructions: http://importantfish.com/how-to-install-hadoop-on-mac-os-x/ That worked like a charm and I'm able to start hadoop and run a basic test: hadoop-examples-1.1.2.jar pi 10 100 Now, I'm using the python client to handle the HTTP requests to/from webhdfs: http://pythonhosted.org/pywebhdfs/ But I'm stumbling on a

问题 What is the difference between the WebHDFS REST API and HttpFS? If I understand correctly: HttpFS is an independent service that exposes a REST API on top of HDFS WebHDFS is a REST API built-into HDFS. It doen't require any further installation Am I correct? When would be advisable to use one instead of the other? 回答1: I have read a article related with your question. following is the link. https://www.linkedin.com/today/post/article/20140717115238-176301000-accessing-hdfs-using-the-webhdfs

问题 What is the difference between the WebHDFS REST API and HttpFS? If I understand correctly: HttpFS is an independent service that exposes a REST API on top of HDFS WebHDFS is a REST API built-into HDFS. It doen't require any further installation Am I correct? When would be advisable to use one instead of the other? 回答1: I have read a article related with your question. following is the link. https://www.linkedin.com/today/post/article/20140717115238-176301000-accessing-hdfs-using-the-webhdfs

问题 I want to access videos which are stored in HDFS & show them in a web browser. I found this project ( https://github.com/yeleaveszi/Play-Videos-In-HDFS ) on github which exactly does this, but couldn't make it work. Can anyone provide me some information or ways in which I could show the video stored in HDFS on a web browser. Also is it possible to get a HTTP link of the video which is stored in HDFS, I read about WebHDFS & HTTPFS but didn't understand much. Any help would be appreciated.

问题 I am doing HttpPut with MultiPartEntity to write a file to HDFS via the webHDFS REST API. The request itself goes through and gives me the right responses, 307 and 201. However the image has multipart headers also written as part of it as shown below and its not a valid image to retrieve and open. --8DkJ3RkUHahEaNE9Ktw8NC1TFOqegjfA9Ps Content-Disposition: form-data; name="file"; filename="advert.jpg" Content-Type: application/octet-stream ÿØÿàJFIFHHÿÛC // Rest of the image content -