NoClassDefFoundError与gradle，giraph和hadoop

Question

So, I've been looking around a lot and I haven't found a good answer to my question, and this is driving me crazy, so I figured I'd ask here and hopefully I can get help. I'm trying to do automated testing in a Giraph project using gradle. I'm a total beginner with gradle. Just to get started, I copied the test code for the SimpleShortestPathComputation class into my project, to make sure I could get the tests up and running. When I do gradle test , however, I get the following error:

$ gradle test --info
<skipping some output here...>

Successfully started process 'Gradle Test Executor 1'
Gradle Test Executor 1 started executing tests.
WCCTest > testToyData FAILED
    java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:142)
        at WCCTest.testToyData(WCCTest.java:180)

    Caused by:
    java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        ... 2 more

Gradle Test Executor 1 finished executing tests.
WCCTest > testOnShorterPathFound FAILED
    java.lang.NoClassDefFoundError: org.apache.hadoop.conf.Configuration
        at sun.reflect.GeneratedSerializationConstructorAccessor33.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at     org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:59)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:128)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:63)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:56)
        at org.mockito.internal.creation.CglibMockMaker.createMock(CglibMockMaker.java:23)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:26)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:51)
        at org.mockito.Mockito.mock(Mockito.java:1243)
        at org.mockito.Mockito.mock(Mockito.java:1120)
        at org.apache.giraph.utils.MockUtils$MockedEnvironment.<init>(MockUtils.java:68)
        at org.apache.giraph.utils.MockUtils.prepareVertexAndComputation(MockUtils.java:132)
        at WCCTest.testOnShorterPathFound(WCCTest.java:64)

WCCTest > testToyDataJson FAILED
    java.lang.NoClassDefFoundError: Could not initialize class org.apache.giraph.conf.GiraphConfiguration
        at WCCTest.testToyDataJson(WCCTest.java:127)

WCCTest > testOnNoShorterPathFound FAILED
    java.lang.NoClassDefFoundError: org.apache.hadoop.conf.Configuration
        at sun.reflect.GeneratedSerializationConstructorAccessor33.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:59)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:128)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:63)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:56)
        at org.mockito.internal.creation.CglibMockMaker.createMock(CglibMockMaker.java:23)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:26)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:51)
        at org.mockito.Mockito.mock(Mockito.java:1243)
        at org.mockito.Mockito.mock(Mockito.java:1120)
        at org.apache.giraph.utils.MockUtils$MockedEnvironment.<init>(MockUtils.java:68)
        at org.apache.giraph.utils.MockUtils.prepareVertexAndComputation(MockUtils.java:132)
        at WCCTest.testOnNoShorterPathFound(WCCTest.java:95)

4 tests completed, 4 failed
<more output...>
:test FAILED
:test (Thread[main,5,main]) completed. Took 1.663 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///...build/reports/tests/index.html

BUILD FAILED

I'm using the totally standard project directory structure, and this is my build.gradle file:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    compile files('$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar')
    compile files('$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar')
    compile files('$HADOOP_HOME/hadoop-core-0.20.203.0.jar')

    testCompile group: 'junit', name: 'junit', version: '4.+'
    testCompile group: 'org.mockito', name: 'mockito-all', version: '1.9.5'
}

It compiles with no problem, and the jar files I'm including as dependences include the classes for which it says NoClassDefFoundError (according to jar tf ). Any ideas what I'm doing wrong? Thanks in advance.

Answer 1

try add following into your build.gradle

    println("HADOOP_HOME=$HADOOP_HOME")
    compile files("$HADOOP_HOME/hadoop-core-0.20.203.0.jar")
    println("System.env.HADOOP_HOME=$System.env.HADOOP_HOME")
    compile files("$System.env.HADOOP_HOME/hadoop-core-0.20.203.0.jar")

Answer 2

It turned out I just had to add some of the jars from the $HADOOP_HOME/lib directory to the dependencies and it worked. Once I added the dependency for org/apache/commons/logging/LogFactory it cleared up the error for the hadoop Configuration class, and then I just had to add the other required jars later.