Solr, clustering (carrot) and NoClassDefFoundError

Question

i'm running Solr 3.4 and would like to use the clusteringComponent. Following this tutorial: http://wiki.apache.org/solr/ClusteringComponent in combination with default entries at the solrconfig.xml i have the following configuration @solrconfig.xml

  <searchComponent name="clustering" 
                   enable="${solr.clustering.enabled:true}"
                   class="org.apache.solr.handler.clustering.ClusteringComponent" >

    <!-- Declare an engine -->
    <lst name="engine">
      <str name="name">default</str>
      <str name="carrot.algorithm">org.carrot2.clustering.lingo.LingoClusteringAlgorithm</str>
      <str name="LingoClusteringAlgorithm.desiredClusterCountBase">20</str>
    </lst>

    <lst name="engine">
      <str name="name">stc</str>
      <str name="carrot.algorithm">org.carrot2.clustering.stc.STCClusteringAlgorithm</str>
    </lst>
  </searchComponent>


<requestHandler name="/cl" class="solr.SearchHandler" >
  <lst name="defaults">
    <str name="echoParams">explicit</str>

    <bool name="clustering">true</bool>
    <str name="clustering.engine">default</str>
    <bool name="clustering.results">true</bool>

    <!-- Fields to cluster on -->
    <str name="carrot.title">UEBSCHRIFT</str>
    <str name="carrot.snippet">TEXT</str>
  </lst>

So if i try to use the requestHandler http://server:8080/solr/mycore/cl?q=*:* i get the following Java exception:

java.lang.NoClassDefFoundError: com.carrotsearch.hppc.ObjectContainer
    at java.lang.J9VMInternals.verifyImpl(Native Method)
    at java.lang.J9VMInternals.verify(J9VMInternals.java:72)
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:134)
    at org.carrot2.text.preprocessing.pipeline.BasicPreprocessingPipeline.<init>(BasicPreprocessingPipeline.java:106)
    at org.carrot2.text.preprocessing.pipeline.CompletePreprocessingPipeline.<init>(CompletePreprocessingPipeline.java:32)
    at org.carrot2.clustering.lingo.LingoClusteringAlgorithm.<init>(LingoClusteringAlgorithm.java:129)
    at java.lang.J9VMInternals.newInstanceImpl(Native Method)
    at java.lang.Class.newInstance(Class.java:1325)
    at org.carrot2.util.pool.SoftUnboundedPool.borrowObject(SoftUnboundedPool.java:80)
    at org.carrot2.core.PoolingProcessingComponentManager.prepare(PoolingProcessingComponentManager.java:128)
    at org.carrot2.core.Controller.process(Controller.java:333)
    at org.carrot2.core.Controller.process(Controller.java:240)
    at org.apache.solr.handler.clustering.carrot2.CarrotClusteringEngine.cluster(CarrotClusteringEngine.java:136)
    at org.apache.solr.handler.clustering.ClusteringComponent.process(ClusteringComponent.java:91)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:735)
Caused by: java.lang.ClassNotFoundException: com.carrotsearch.hppc.ObjectContainer
    at java.lang.Throwable.<init>(Throwable.java:80)
    at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:76)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:419)
    at java.lang.ClassLoader.loadClass(ClassLoader.java)
    at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:923)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:609)
    ... 31 more

The point is, that I have no idea what this mean. I'm searching for hours without finding an solution.

By the way: I'm running on tomcat with the following options:

export CATALINA_OPTS="-Dsolr.clustering.enabled=true"

(is this still required in Solr 3.4?)

The catalina option is part of the java command, as you can see with ps -efa

/usr/lib64/jvm/java-1_6_0-ibm-1.6.0/jre//bin/java -Djava.util.logging.config.file=/opt/tomcat6/conf/logging.properties -Xms2048m -Xmx2048m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dsolr.clustering.enabled=true -Djava.endorsed.dirs=/opt/tomcat6/endorsed -classpath /opt/tomcat6/bin/bootstrap.jar -Dcatalina.base=/opt/tomcat6 -Dcatalina.home=/opt/tomcat6 -Djava.io.tmpdir=/opt/tomcat6/temp org.apache.catalina.startup.Bootstrap start

Does anyone has an idea what i could do to solve this problem?

//Update: if i add hppc-0.3.4-jdk15.jar, i get the following error:

java.lang.NoClassDefFoundError: org.apache.mahout.math.matrix.DoubleMatrix2D
    at java.lang.J9VMInternals.verifyImpl(Native Method)
    at java.lang.J9VMInternals.verify(J9VMInternals.java:72)
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:134)
    at org.carrot2.clustering.lingo.LingoClusteringAlgorithm.<init>(LingoClusteringAlgorithm.java:134)
[...]
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.math.matrix.DoubleMatrix2D
    at java.lang.Throwable.<init>(Throwable.java:80)
    at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:76)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:419)
    at java.lang.ClassLoader.loadClass(ClassLoader.java)
    at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:923)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:609)
    ... 29 more

it looks like, i have to install an mahout archive, but i think, all packages for clustering are included in Solr 3.4?! It looks like, i'm on the wrong way?!

Answer 1

If you are using Solr with tomcat as a seperate instance, you would need to copy the jars so that they are available for Solr.

Quote from README.txt

NOTE: This Solr example server references certain Solr jars outside of this server directory for non-core modules with statements in solrconfig.xml. If you make a copy of this example server and wish to use the ExtractingRequestHandler (SolrCell), DataImportHandler (DIH), UIMA, the clustering component, or other modules in "contrib", you will need to copy the required jars into solr/lib or update the paths to the jars in your solrconfig.xml.

Check for the clustering and carrot jars in solrconfig.xml.
Probably you are missing hppc-0.3.4-jdk15.jar

Answer 2

Did you add the Mahout Math package?

It seems to be a separate package.

NoClassDefFoundError: org.apache.mahout.math.matrix.DoubleMatrix2D
                      ^^^^^^^^^^^^^^^^^^^^^^

Answer 3

Why not use Solr's default packaging (this is officially supported)? It ships with Jetty and will save you the headaches connected with classpath because things are already configured.

Answering your question, you'll need all the JARs from Solr's default clustering extension folders; for 4.0 alpha this would be: contrib/clustering/lib/*.jar

carrot2-core-3.5.0.jar hppc-0.3.3.jar jackson-core-asl-1.7.4.jar jackson-mapper-asl-1.7.4.jar mahout-collections-0.3.jar mahout-math-0.3.jar simple-xml-2.4.1.jar

Answer 4

With solR 4 copy this file in folder conf

http://svn.apache.org/repos/asf/labs/alike/trunk/demo/solrhome/collection1/conf/solrconfig.xml