Installing Hadoop's Eclipse Plugin

Question

I have such a nightmare with the Hadoop\'s Eclipse plugin. First of all the latest Hadoop version (which is 1.1.1) doesn\'t even include the Eclipse plugin jar file. version 1.0

Answer 1

I installed eclipse plugin for hadoop-1.0.4 yesterday with difficulty, and tested it successful.

The reason for plugin not working is that the jar package lost some libs :

• commons-cli-1.2.jar
• commons-configuration-1.6.jar
• jackson-core-asl-1.8.8.jar
• jackson-mapper-asl-1.8.8.jar
• commons-httpclient-3.0.1.jar
• commons-lang-2.4.jar

you could cp this jars from ${hadoop}/lib to ${jar}/lib, and don't forget modifying MANIFEST.

For convenience, I add some code to ${eclipse-plugin-src}/build.xml target jar

<copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-${commons-configuration.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-${commons-httpclient.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-${commons-lang.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

and modified MANIFEST.MF in ${eclipse-plugin-src}/META-INF

undle-ClassPath: classes/,
 lib/hadoop-core.jar,
 lib/commons-cli-1.2.jar,
 lib/commons-configuration-1.6.jar,
 lib/jackson-core-asl-1.8.8.jar,
 lib/commons-httpclient-3.0.1.jar,
 lib/jackson-mapper-asl-1.8.8.jar,
 lib/commons-lang-2.4.jar

rebuild eclipse-plugin, and HAVE FUN!

Answer 2

Well, combined with hs3180's answer, here is what I actually did in configuring the compiling.

1. In ${eclipse-plugin-src}/build.xml, use the following directives to include required jars by the eclipse-plugin. Note here that, instead of from ${hadoop.root}/build, copying of the hadoop-core-${version}.jar is now from ${hadoop.root} directly, because if you are using the compiled version of hadoop, the ${hadoop.root}/build folder would be actually empty. Copying of the commons-cli-${commons-cli.version}.jar is now from ${hadoop.root}/lib for the same reason.

   <!-- <copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> -->
   <!-- <copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/> -->
   <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-configuration-${commons-configuration.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-httpclient-${commons-httpclient.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-lang-${commons-lang.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/jackson-core-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/jackson-mapper-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   

2. Modify the ${eclipse-plugin-src}/META-INF/MANIFEST.MF file to accommodate the changes made to ${eclipse-plugin-src}/build.xml.

   Bundle-ClassPath: classes/,
    lib/hadoop-core.jar,
    lib/commons-cli-1.2.jar,
    lib/commons-configuration-1.6.jar,
    lib/jackson-core-asl-1.8.8.jar,
    lib/commons-httpclient-3.0.1.jar,
    lib/jackson-mapper-asl-1.8.8.jar,
    lib/commons-lang-2.4.jar
   

3. Modify the ${hadoop.root}/src/contrib/build-contrib.xml file at two places. The first is to set up the properties needed for the plugin compiling. And the second is to make sure jars, especially hadoop-core-1.0.4.jar, in the ${hadoop.root} folder are visible to javac, because the eclipse-plugin references the hadoop classes. This setup is different from the first two in purpose and thus not dispensable.

   <!-- Properties added for compiling eclipse-plugin -->
   <!-- http://yiyujia.blogspot.com/2012/11/build-hadoop-eclipse-plugin-from-source.html -->
   <property name="eclipse.home" location="/Users/xuj/Downloads/eclipse/"/>
   <property name="version" value="1.0.4"/>
   <property name="commons-cli.version" value="1.2"/>
   
   <!-- the normal classpath -->
   <path id="contrib-classpath">
     <fileset dir="${hadoop.root}">
       <include name="*.jar" />
     </fileset>
     <!-- more path elements go here -->
   </path>
   

When all files correctly configured, call 'ant jar' from ${eclipse-plugin-src}/ in the console shall be sufficient for the rest.