how can I compile/using mahout for hadoop 2.0?

Question

The latest release mahout 0.9 is only built on hadoop 1.x. (mvn clean install) How can I compile mahout for hadoop 2.0.x?

Because When I was running the commands:

hadoop jar mahout-examples-0.9-SNAPSHOT-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -s SIMILARITY_COOCCURENCE -i test -o result

I always got the error message

IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected.

Thanks!

Answer 1

To compile Mahout to work with 2.x since it isn't released in a package that is compatible with Hadoop 2.x:

mvn clean install -Dhadoop2 -Dhadoop2.version=2.2.0 -DskipTests=true

If you want to confirm after the build that the correct dependencies were brought in, run the following from the root of the project:

find . -name hadoop*.jar

The artifacts generated by the above command differ from what's in the 0.9 release so you'll need to update more than just the version number (there seems to be a new mahout-mrlegacy jar).

That being said, if someone can point me to an official artifact that runs with 2.x (even if it's a SNAPSHOT), I'd appreciate it.

Answer 2

Did you try changing the version of hadoop in the pom.xml?

<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>                                                   
<version> 0.23.9</version> 

(Please check the version of hadoop. As I understand, 0.23 became 2.0)

Another thing to try is to run the RecommenderJob directly like this:

bin/mahout recommenditembased \
   --input INPUT --output OUTPUT \
   --tempDir TEMP --similarityClassname SIMILARITY_LOGLIKELIHOOD

To run this you have to be in the root of your mahout folder, and you need to have the following environment variables set:

export HADOOP_HOME=/path/to/hadoop/home
export HADOOP_COMMONS_HOME=/path/to/commons/home
export JAVA_HOME=/path/to/java/home
export PATH="$PATH:$HADOOP_HOME/bin"

Answer 3

About an hour ago Mahout has officially added support to Hadoop 2.x on the master branch (see MAHOUT-1329)

Checkout the code here https://github.com/apache/mahout and recompile using:

mvn clean package -Dhadoop2.version=2.2.0

Try and see if that works.