How can I run Hadoop run with a Java class?

前端 未结 9 806
甜味超标
甜味超标 2021-01-20 03:13

I am following the book Hadoop: the definitive Guide.

I am confused on example 3-1.

There is a Java source file, URLCat.java. I use javac to co

相关标签:
9条回答
  • 2021-01-20 03:59

    Go to the directory where your compiled .class files are residing.

    Use full class name including package name (refer to Receiving "wrong name" NoClassDefFoundError when executing a Java program from the command-line for full class name or which directory to run the job in) when running hadoop URLCat hdfs://localhost/user/tom/quangle.txt.

    In my case URLCat.java was in com.tom.app, so the hadoop command was hadoop com.tom.app.URLCat hdfs://localhost/user/tom/quangle.txt.

    0 讨论(0)
  • 2021-01-20 04:00

    We can access HDFS through the hdfs api. My understanding of it is that you can use the hdfs api to contact a hadoop cluster running the dfs and fetch data from it.

    Why do we need to invoke the command as hadoop jar URLCat.jar

    why not just java URLCat

    Why does the client necessarily need to install hadoop and then contact the hadoop cluster?

    0 讨论(0)
  • 2021-01-20 04:09

    Of course you could use cat, but that sort of isn't the point (i.e. you're learning, not just trying to get it to work).

    As per the book, you need to set your HADOOP_CLASSPATH environment variable. In my case, using the build example in the book, all of my classes are at: /media/data/hadefguide/book/build/classes

    Here's an example:

    hduser@MuleBox ~ $ export HADOOP_CLASSPATH=
    hduser@MuleBox ~ $ hadoop URLCat hdfs://localhost/user/hduser/quangle.txt
    Exception in thread "main" java.lang.NoClassDefFoundError: URLCat
    Caused by: java.lang.ClassNotFoundException: URLCat
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    Could not find the main class: URLCat.  Program will exit.
    hduser@MuleBox ~ $ export HADOOP_CLASSPATH=/media/data/hadefguide/book/build/classes
    hduser@MuleBox ~ $ hadoop URLCat hdfs://localhost/user/hduser/quangle.txt
    On the top of the Crumpetty Tree
    The Quangle Wangle sat,
    But his face you could not see,
    On account of his Beaver Hat.
    
    0 讨论(0)
提交回复
热议问题