Spring Boot胖jar中的NoClassDefFoundError，但依赖是在编译的存档中，应用程序在IDE中运行正常

Question

UPDATE: SOLVED

For some reason, the class loader that I was using to load my custom class, which extends GenModel , did not have the context of the classes that were already loaded, which I did not anticipate. So, even though GenModel was loaded from my debugging perspective, it was not loaded from my modelLoader 's perspective.

Solution : URLClassLoader instances can be handed a parent context. Not knowing exactly which context to hand it, I just retrieved the context of the GenModel class I knew to be already loaded, figuring that that context would obviously contain GenModel . So, in the end I only changed one line of code:

URLClassLoader modelLoader = new URLClassLoader(new URL[]{modelURL});

became

URLClassLoader modelLoader = new URLClassLoader(new URL[]{modelURL},GenModel.class.getClassLoader());

and that solved it!

Shout out to @EvilToad, who put me on the right track through his comments!

ORIGINAL ISSUE:

I am building a prototype that implements compiled h2o neural network models as web services, injecting the model at runtime. The app runs in a Spring Boot embedded Tomcat server and is a Maven project. I have the following dependency in the Maven POM:

<dependency>
  <groupId>ai.h2o</groupId>
  <artifactId>h2o-genmodel</artifactId>
  <version>3.8.2.3</version>
</dependency>

This dependency includes the class hex.genmodel.GenModel . When the application starts, it loads the custom model class using a URLClassLoader and casts it as a GenModel . The relevant code is as follows:

import hex.genmodel.GenModel;

...

@Bean
public NeuralNetPredictor neuralNetPredictor() throws InstantiationException, IllegalAccessException, ClassNotFoundException, IOException {
    URL modelURL = (new File(this.modelJarPath)).toURI().toURL();
    URLClassLoader modelLoader = new URLClassLoader(new URL[]{modelURL});
    GenModel rawModel = (GenModel) modelLoader.loadClass(this.modelClassName).newInstance();
    modelLoader.close();

    return new NeuralNetPredictor(rawModel, this.modelClassName);
}

And here is my POM:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>TVGML</groupId>
    <artifactId>neuralnetpredictor</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>1.3.5.RELEASE</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>ai.h2o</groupId>
            <artifactId>h2o-genmodel</artifactId>
            <version>3.8.2.3</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <version>1.3.5.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger2</artifactId>
            <version>2.2.2</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger-ui</artifactId>
            <version>2.2.2</version>
        </dependency>
    </dependencies>
    <properties>
        <java.version>1.8</java.version>
    </properties>
</project>

When I make and run the application using the following IntelliJ configuration, everything runs great:

IntelliJ run config

Everything loads, the server comes up as expected, all functionality works, etc.

On the other hand, if I do a mvn clean install or package that also builds fine, but when I run java -jar <myjarartifact> , the application throws an exception:

... Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.----.research.machinelearning.neuralnets.NeuralNetPredictor]: Factory method 'neuralNetPredictor' threw exception; nested exception is java.lang.NoClassDefFoundError: hex/genmodel/GenModel Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.----.research.machinelearning.neuralnets.NeuralNetPredictor]: Factory method 'neuralNetPredictor' threw exception; nested exception is java.lang.NoClassDefFoundError: hex/genmodel/GenModel

... Caused by: java.lang.NoClassDefFoundError: hex/genmodel/GenModel

... Caused by: java.lang.ClassNotFoundException: hex.genmodel.GenModel

Here's the part that throws me off: when I look inside the fat jar produced by spring boot, hex.genmodel.GenModel is there, no different from the other dependencies:

GenModel class in fat jar dependencies

UPDATED: Here's the stack trace

Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:62)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54)
    ... 1 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '/model': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.------.research.machinelearning.neuralnets.NeuralNetPredictor com.------.research.machinelearning.neuralnets.NeuralNetPredictorController.pred; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'neuralNetPredictor' defined in com.------.research.machinelearning.neuralnets.NeuralNetService: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.------.research.machinelearning.neuralnets.NeuralNetPredictor]: Factory method 'neuralNetPredictor' threw exception; nested exception is java.lang.NoClassDefFoundError: hex/genmodel/GenModel
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:766)
    at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:361)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1191)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1180)
    at com.------.research.machinelearning.neuralnets.NeuralNetService.main(NeuralNetService.java:34)
    ... 6 more
Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.------.research.machinelearning.neuralnets.NeuralNetPredictor com.------.research.machinelearning.neuralnets.NeuralNetPredictorController.pred; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'neuralNetPredictor' defined in com.------.research.machinelearning.neuralnets.NeuralNetService: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.------.research.machinelearning.neuralnets.NeuralNetPredictor]: Factory method 'neuralNetPredictor' threw exception; nested exception is java.lang.NoClassDefFoundError: hex/genmodel/GenModel
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:573)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
    ... 23 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'neuralNetPredictor' defined in com.------.research.machinelearning.neuralnets.NeuralNetService: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.------.research.machinelearning.neuralnets.NeuralNetPredictor]: Factory method 'neuralNetPredictor' threw exception; nested exception is java.lang.NoClassDefFoundError: hex/genmodel/GenModel
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:545)
    ... 25 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.------.research.machinelearning.neuralnets.NeuralNetPredictor]: Factory method 'neuralNetPredictor' threw exception; nested exception is java.lang.NoClassDefFoundError: hex/genmodel/GenModel
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    ... 37 more
Caused by: java.lang.NoClassDefFoundError: hex/genmodel/GenModel
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.------.research.machinelearning.neuralnets.NeuralNetService.neuralNetPredictor(NeuralNetService.java:46)
    at com.------.research.machinelearning.neuralnets.NeuralNetService$$EnhancerBySpringCGLIB$$352d44ca.CGLIB$neuralNetPredictor$0(<generated>)
    at com.------.research.machinelearning.neuralnets.NeuralNetService$$EnhancerBySpringCGLIB$$352d44ca$$FastClassBySpringCGLIB$$21caa282.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
    at com.------.research.machinelearning.neuralnets.NeuralNetService$$EnhancerBySpringCGLIB$$352d44ca.neuralNetPredictor(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
    ... 38 more
Caused by: java.lang.ClassNotFoundException: hex.genmodel.GenModel
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 60 more

Any insight is greatly appreciated!

Answer 1

URLClassLoader instances can be handed a parent context. Not knowing exactly which context to hand it, I just retrieved the context of the GenModel class I knew to be already loaded, figuring that that context would obviously contain GenModel. So, in the end I only changed one line of code:

URLClassLoader modelLoader = new URLClassLoader(new URL[]{modelURL});

became

URLClassLoader modelLoader = new URLClassLoader(new URL[]{modelURL},GenModel.class.getClassLoader());

and that solved it!

Shout out to @EvilToad, who put me on the right track through his comments!