Cloudera Quickstart VM illegalArguementException: Wrong FS: hdfs: expected: file:

问题

I have a simple java code to copy a text file from my local to the hdfs. I am using cloudera's quickstart virtual machine.

Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/cloudera/workspace/Downloader/output/data.txt"), 
                          new Path("hdfs://quickstart.cloudera:8020/user/cloudera/"));

I get this error after running this code:

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://quickstart.cloudera:8020/user/cloudera, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at org.apache.hadoop.fs.LocalFileSystem.pathToFile(LocalFileSystem.java:61)
at org.apache.hadoop.fs.LocalFileSystem.exists(LocalFileSystem.java:51)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:355)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:211)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:67)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1143)

What could I be doing wrong?

回答1:

I resolved this problem. You have to be careful with the kind of jar files you add to your classpath especially when working with cloudera quickstart vm. If available, use jar files provided by cloudera. They can be found in this folder: /usr/lib/hadoop/client/ This code will work fine without any problems.

Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/cloudera/workspace/Downloader/output/data.txt"), 
                      new Path("hdfs://quickstart.cloudera:8020/user/cloudera/"));

回答2:

Not correct path("hdfs://quickstart.cloudera:8020/user/cloudera/" use this example:

Configuration conf = getConf();
System.out.println("fs.default.name : - " + conf.get("fs.default.name"));
// It prints uri  as : hdfs://10.214.15.165:9000 or something...
String uri = conf.get("fs.default.name");
FileSystem fs = FileSystem.get(uri,getConf());

来源：https://stackoverflow.com/questions/33267367/cloudera-quickstart-vm-illegalarguementexception-wrong-fs-hdfs-expected-file

标签

java

eclipse

Hadoop

cloudera-quickstart-vm