奇怪的Tomcat 8部署行为

Question

When I deploy my Spring 4.0.1.RELEASE web application to Tomcat 8.0.33 installed on ext3 running on Java 1.8.0_92 on Centos 6.7 I get the following error:

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'qualityAuditTokenService' defined in URL [jar:file:/home/www/webapps/ROOT/WEB-INF/lib/product-service-2.0.1-SNAPSHOT.jar!/com/company/product/services/QualityAuditTokenService.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [com.company.workflow.dao.TokenDao]: : No qualifying bean of type [com.company.workflow.dao.TokenDao] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {}; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [com.company.workflow.dao.TokenDao] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {}
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:742)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:196)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1114)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1017)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:700)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4811)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5251)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1092)
    at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1834)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [com.company.workflow.dao.TokenDao] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {}
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoSuchBeanDefinitionException(DefaultListableBeanFactory.java:1100)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:960)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:855)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:806)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:734)
    ... 28 more

However if I install exactly the same web application to Tomcat 8.0.33 installed on an NFS mount running on Java 1.8.0_92 on Centos 6.7 it works just fine. It also works just fine if I install it to Tomcat 7.0.69 on ext3 on Centos, Tomcat 8.0.33 on ext4 on Ubuntu and NTFS on Windows. So it's just throwing this error running in Tomcat 8.0.33 on ext3 on Centos. It wouldn't be so much of a problem if this weren't our live deployment environment.

So this is clearly not one of the standard "missing annotations" or "bean class missing from JAR" type problems although I am happy to hear suggestions in this vein in case I missed something.

The strange thing about this deployment is that the Spring beans are created in a different order on the different file systems. In the versions that work, the following appears in the logfile with Spring logging maxed:

DEBUG DefaultListableBeanFactory:449 - Creating instance of bean 'tokenDaoHbm'
DEBUG DefaultListableBeanFactory:249 - Returning cached instance of singleton bean 'sessionFactory'
DEBUG DefaultListableBeanFactory:249 - Returning cached instance of singleton bean 'searchSessionFactory'
DEBUG DefaultListableBeanFactory:750 - Autowiring by type from bean name 'tokenDaoHbm' via constructor to bean named 'sessionFactory'
DEBUG DefaultListableBeanFactory:523 - Eagerly caching bean 'tokenDaoHbm' to allow for resolving potential circular references
DEBUG DefaultListableBeanFactory:249 - Returning cached instance of singleton bean 'org.springframework.transaction.config.internalTransactionAdvisor'
DEBUG DefaultListableBeanFactory:249 - Returning cached instance of singleton bean 'org.springframework.cache.config.internalCacheAdvisor'
DEBUG AnnotationTransactionAttributeSource:108 - Adding transactional method 'TokenDaoHbm.update' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT; ''
DEBUG InfrastructureAdvisorAutoProxyCreator:551 - Creating implicit proxy for bean 'tokenDaoHbm' with 0 common interceptors and 1 specific interceptors
DEBUG JdkDynamicAopProxy:117 - Creating JDK dynamic proxy: target source is SingletonTargetSource for target object [com.company.product.dao.hibernate.TokenDaoHbm@4a51d9f9]
DEBUG DefaultListableBeanFactory:477 - Finished creating instance of bean 'tokenDaoHbm'

This is the bean that would satisfy the dependency had it been created - in the version that throws the exception, this bean creation is notable for it's absence.

TL;DR

So, how can the OS, file system type and/or network latency change the order which Spring creates beans (or otherwise break it's dependency analysis)? Surely this is something enshrined in the WAR file (and the version of Spring it is packaged with)?

I have tried to influence bean creation via @ComponentScan and @Qualifier to no avail - are there other approaches?

This problem bears a resemblance to the one linked below, but there is no posted solution (and they are having the problem with Tomcat 7 not 8). Need help debugging Tomcat 7 application error

Any help with this greatly appreciated, as this one is really vexing me! :-D

Answer 1

I now have a solution to this (hence posting an answer) but it's butt fugly and I still have no explanation for why this is necessary (so I'll save the accepted tick for either a more elegant solution or a complete explanation).

It turns out my issue is related to issue 57129 raised on the ASF bugzilla: https://bz.apache.org/bugzilla/show_bug.cgi?id=57129

However, in the case outlined there, multiple JAR files within a WAR contain different versions of the same class file. This means changing the order will change the application behaviour - not desirable.

In my case, the class in question, TokenDaoHbm, only exists once in the WAR file. It's just that, if the Tomcat class loader hasn't loaded the product-dao-hibernate-2.0.1-SNAPSHOT JAR file by the time Spring comes to instantiate qualityAuditTokenService bean, then you get a NoSuchBeanDefinitionException. Surely Spring and/or Tomcat must know all classes must be loaded before bean instantiation can commence?

So, to fix my problem, I placed the following in my application context.xml in the WAR according to the advice from Mark Thomas in the ASF bug posting:

<Resources>
  <PreResources className="org.apache.catalina.webresources.FileResourceSet"
                base="${catalina.base}/webapps/ROOT/WEB-INF/lib/product-dao-hibernate-2.0.1-SNAPSHOT.jar"
                webAppMount="/WEB-INF/lib/product-dao-hibernate-2.0.1-SNAPSHOT.jar" />
</Resources>

If anyone can shed further light on this I am happy to mark them as the accepted answer.