Oracle BLOB exception during deployment/starting up war on Wildfly8

Question

I have following exception during start-up of war java /w spring app

Caused by: java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.scheduling.quartz.SchedulerFactoryBean#0' defined in "/content/service.war/WEB-INF/classes/META-INF/spring/applicationContext-service.xml": Invocation of init method failed; nested exception is org.quartz.JobPersistenceException: Couldn't retrieve job because the BLOB couldn't be deserialized: null [See nested exception: java.io.EOFException]
    at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:222)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:87)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:72)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
    ... 3 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.scheduling.quartz.SchedulerFactoryBean#0' defined in "/content/service.war/WEB-INF/classes/META-INF/spring/applicationContext-service.xml": Invocation of init method failed; nested exception is org.quartz.JobPersistenceException: Couldn't retrieve job because the BLOB couldn't be deserialized: null [See nested exception: java.io.EOFException]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:615)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
    at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173)
    at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:193)
    ... 7 more
Caused by: org.quartz.JobPersistenceException: Couldn't retrieve job because the BLOB couldn't be deserialized: null [See nested exception: java.io.EOFException]
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1421)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$7.execute(JobStoreSupport.java:1396)
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:242)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeWithoutLock(JobStoreSupport.java:3693)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393)
    at org.quartz.core.QuartzScheduler.getJobDetail(QuartzScheduler.java:1431)
    at org.quartz.impl.StdScheduler.getJobDetail(StdScheduler.java:539)
    at org.springframework.scheduling.quartz.SchedulerAccessor.jobDetailExists(SchedulerAccessor.java:420)
    at org.springframework.scheduling.quartz.SchedulerAccessor.addJobToScheduler(SchedulerAccessor.java:338)
    at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:281)
    at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:508)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)
    ... 21 more
Caused by: java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2328) [rt.jar:1.8.0_45]
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797) [rt.jar:1.8.0_45]
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) [rt.jar:1.8.0_45]
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) [rt.jar:1.8.0_45]
    at org.quartz.impl.jdbcjobstore.oracle.OracleDelegate.getObjectFromBlob(OracleDelegate.java:156)
    at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:904)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1404)
    ... 33 more

It is deployed on clustered wildfly 8.1.x. The application uses Quartz ver 1.8.6. The Spring version is 3.2.5. I had similar (no EOF mentioned) problem on lower enviroment, but it was due to lack of JDBC driver configuration.

The quartz configuration is similar to this one ( http://quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigJDBCJobStoreClustering ), except the Data Source is configured on the wildfly.

The DS itslef bounds correctly:
2015-09-25 14:57:21,411 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) JBAS010400: Bound data source [java:/quartzDs]

I am really confused with this, because on another (mirrored) enviroment it works just fine.

Any clues?

Answer 1

Okay, I've manage to resolve the issuse. It turned out that the problem is with org.quartz.impl.jdbcjobstore.oracle.OracleDelegate and Oracle 10.2.x. It works fine with Oracle 11.x. The problem itself is that the jobs in Oracle 10.2.x are serialized as null therefore they can't be deserialized later, hence the above stack trace.

To resolve this issue I had write own delegate extending OracleDelegate and override getObjectFromBlob method.

public class CustomDelegate extends OracleDelegate {

    public CustomDelegate(Logger logger, String tablePrefix, String instanceId) {
        super(logger, tablePrefix, instanceId);
    }

    public CustomDelegate(Logger logger, String tablePrefix, String instanceId, Boolean useProperties) {
        super(logger, tablePrefix, instanceId, useProperties);
    }

    protected Object getObjectFromBlob(ResultSet rs, String colName)
            throws ClassNotFoundException, IOException, SQLException {
        byte[] data = rs.getBytes(colName);
        if (data == null || data.length == 0) {
            return null;
        }
        ObjectInputStream in = new ObjectInputStream(new ByteArrayInputStream(data));
        try {
            return in.readObject();
        } finally {
            in.close();
        }
    }
}

Then I had to change the quartz properties

org.quartz.jobStore.driverDelegateClass = my.package.CustomDelegate

Now it works fine.

Answer 2

I was also facing similar issue and was getting this error while server start.

org.quartz.JobPersistenceException: Couldn't retrieve job because the BLOB couldn't be deserialized: null
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1397) ~[quartz-2.3.0.jar:na]

I cleared data from QRTZ_JOB_DETAILS table for the job I was getting error and restarted my server. This fixed the problem.

delete from QRTZ_JOB_DETAILS where job_name = 'MergeEntitiesJob';
commit;

This issue happens when BLOB data contained in JOB_DATA column of QRTZ_JOB_DETAILS gets corrupted. Clearing the row and restarting the application fixes this problem. Reference: http://quartz.10975.n7.nabble.com/Couldn-t-retrieve-job-because-the-BLOB-couldn-t-be-deserialized-Error-td5755.html

I hope this would help.