How can I run Hadoop run with a Java class?

前端 未结 9 796
甜味超标
甜味超标 2021-01-20 03:13

I am following the book Hadoop: the definitive Guide.

I am confused on example 3-1.

There is a Java source file, URLCat.java. I use javac to co

相关标签:
9条回答
  • 2021-01-20 03:46

    step 1: Compile Java Program:

    javac URLCat.java -classpath $HADOOP_HOME/share/hadoop/common/hadoop-common-2.7.0.jar

    step 2: Create jar file :

    jar cvf URLCat.jar URLCat.class

    Step 3: Execute program : (mention your hdfs file location)

    hadoop jar URLCat.jar URLCat hdfs://localhost:9000/pcode/wcinput.txt

    0 讨论(0)
  • 2021-01-20 03:49

    Not sure how Useful is the answer now. I faced the same issue today in fact working on an example from the same book (Hadoop definitive guide) I was able to execute an example program as follows:

    • Write your java code and save it as .java file

    • Compile your java program using:

      javac -classpath <path to hadoop core and commons-cli jar file> <path to your java program file>
      
    • Create a jar file containing your class file:

      jar cvf <jar file> <class files to add separated by space>
      
    • Execute the jar file using hadoop command line:

      hadoop jar <jar file name> <class name containing your main method> <argument to the main method>
      

      e.g.

      hadoop jar FileSystemCat.jar FileSystemCat hdfs://localhost/user/root/MyFiles/meet_a_seer.txt
      

    Hope it helps

    0 讨论(0)
  • 2021-01-20 03:51

    To make the hadoop URLCat command work you need to get the jar (URLCat.jar) to be in your class path. You can put it in lib/ dir of hadoop for that.

    For the hadoop jar URLCat.jar to run you need to create a jar that will have Main class defined in it, otherwise it thinks that the next argument on the command line is the class name. What you can try is hadoop jar URLCat.jar URLCat hdfs://...

    0 讨论(0)
  • 2021-01-20 03:52

    It's quite simple:

    [me@myhost ~]$ hadoop jar
    RunJar jarFile [mainClass] args...
    

    So, what you want is hadoop jar yourJar.jar your.class.with.Main [any args]

    0 讨论(0)
  • 2021-01-20 03:55

    I did this based on help found on this site and the hadoop tutorial.

    mkdir urlcat_classes<br>
    javac -classpath /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar -d     urlcat_classes URLCat.java<br>
    jar -cvf urlcat.jar -C urlcat_classes .<br>
    hadoop jar urlcat.jar no.gnome.URLCat       
    hdfs://localhost/user/claus/sample.txt<br>
    <br>
    no.gnome is from 'package no.gnome;' in URLCat.java.<br><br>
    

    regards
    Claus

    0 讨论(0)
  • 2021-01-20 03:58

    The syntax of the command is a little bit different:

    hadoop fs -cat hdfs:///user/tom/quangle.txt
    

    Do you have hadoop home in your path? can you call hadoop without any parameters?

    0 讨论(0)
提交回复
热议问题