java.lang.NoClassDefFoundError: running Mahout example on a hadoop Cluster

Question

I'm following the Mahout In Action tutorial for kmeans clustring, i use the same code found here : with the same pom.xml also. On my local machine using eclipse every thing works fine, so i build the jar file (clustering-0.0.1-SNAPSHOT.jar) and bring it to the cluster (Hortonworks 2.3) when trying to run it using: hadoop jar clustering-0.0.1-SNAPSHOT.jar com.digimarket.clustering.App (I named my project differently) I get this error:

java.lang.NoClassDefFoundError: org/apache/mahout/common/distance/DistanceMeasure

I know it's a dependency issue, I found questions asked by users who had this issue before but couldn't understand how they solved it. here and here

This is the content of mahout directory in my cluster:

ls /usr/hdp/2.3.4.0-3485/mahout/
bin
conf
doc
lib
mahout-examples-0.9.0.2.3.4.0-3485.jar
mahout-examples-0.9.0.2.3.4.0-3485-job.jar
mahout-integration-0.9.0.2.3.4.0-3485.jar
mahout-math-0.9.0.2.3.4.0-3485.jar
mahout-mrlegacy-0.9.0.2.3.4.0-3485.jar
mahout-mrlegacy-0.9.0.2.3.4.0-3485-job.jar

Thanks.

Answer 1

It looks like you have a dependency that is not available to your code on your cluster.

Based on the pom.xml from that project you should be using:

<properties>
  <mahout.version>0.5</mahout.version>
  <mahout.groupid>org.apache.mahout</mahout.groupid>
</properties>
...
<dependencies>
  <dependency>
    <groupId>${mahout.groupid}</groupId>
    <artifactId>mahout-core</artifactId>
    <version>${mahout.version}</version>
  </dependency>
  ...
</dependencies>

The class org.apache.mahout.common.distance.DistanceMeasure is included in the mahout-core-0.*.jar I have mahout-core-0.7.jar and the class is present in there.

You can download that jar and include it with the -libjars flag or you can put it on the hadoop classpath.