webhdfs

Can anyone please tell me that If I am using java application to request some file upload/download operations to HDFS with Namenode HA setup, Where this request go first? I mean how would client know that which namenode is active? It would be great if you provide some workflow type diagram or something that explains request steps in detail(start to end). If hadoop cluster is configured with HA, then it will have namenode IDs in hdfs-site.xml like this : <property> <name>dfs.ha.namenodes.mycluster</name> <value>namenode1,namenode2</value> </property> Whichever NameNode is started first will

I would like to read a file from HDFS into Spark via httpfs or Webhdfs. Something along the lines of sc.textFile("webhdfs://myhost:14000/webhdfs/v1/path/to/file.txt") or, ideally, sc.textFile("httpfs://myhost:14000/webhdfs/v1/path/to/file.txt") Is there a way to get Spark to read the file over Webhdfs/httpfs? I believe WebHDFS/ HttpFS are like streaming sources to transmit the data over REST-API. Then Spark Streaming can be used to receive the data from the WebHDFS/ HttpFS. According to SPARK-2930 document enhancement request, spark.yarn.access.namenodes should also works for webhdfs / hdfs.

Is there a way to get the last modified times of all dirs and files in hdfs? I want to create page that displays the information, but I have no clue how to go about getting the last mod times all in one .txt file. See if it helps : public class HdfsDemo { public static void main(String[] args) throws IOException { Configuration conf = new Configuration(); conf.addResource(new Path("/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/core-site.xml")); conf.addResource(new Path("/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/hdfs-site.xml")); FileSystem fs = FileSystem.get(conf); System.out.println("Enter

I have a working application for managing HDFS using WebHDFS. I need to be able to do this on a Kerberos secured cluster. The problem is, that there is no library or extension to negotiate the ticket for my app, I only have a basic HTTP client. Would it be possible to create a Java service which would handle the ticket exchange and once it gets the Service ticket to just pass it to the app for use in a HTTP request? In other words, my app would ask the Java service to negotiate the tickets and it would return the Service ticket back to my app in a string or raw string and the app would just

The documentation describing how to connect to a kerberos secured endpoint shows the following: curl -i --negotiate -u : "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=..." The -u flag has to be provided but is ignored by curl. Does the --negotiate option cause curl to look for a keytab that was created beforehand with the kinit command, or will curl prompt for credentials? If it looks for a keytab file, what filename will the command be looking for? Being a once-in-a-while-contributor to curl in that area. Here is what you need to know: curl(1) itself knows nothing about Kerberos and will not