Java Spring Batch using embedded database for metadata and a second database for other data

Question

I'm having a bit of trouble with Spring Batch. I am in this situation: I have to analyze the data inside a file and insert everything in a database (Oracle). The problem I am facing is that when I try to configure the database used to save the data the application crashes and closes. I have this error message:

 :: Spring Boot ::        (v2.0.0.RELEASE)

2018-03-28 17:21:25.033  INFO 9912 --- [           main] c.e.demo.SpringMultiprocessApplication   : Starting SpringMultiprocessApplication on XXXel with PID 9912 (C:\Users\......\target\classes started by XXXEl in C:\Users\......)
2018-03-28 17:21:25.037  INFO 9912 --- [           main] c.e.demo.SpringMultiprocessApplication   : No active profile set, falling back to default profiles: default
2018-03-28 17:21:25.094  INFO 9912 --- [           main] ConfigServletWebServerApplicationContext : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5af97850: startup date [Wed Mar 28 17:21:25 CEST 2018]; root of context hierarchy
2018-03-28 17:21:25.491  INFO 9912 --- [           main] o.s.b.f.xml.XmlBeanDefinitionReader      : Loading XML bean definitions from class path resource [batchjob.xml]
2018-03-28 17:21:25.879  WARN 9912 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.stepScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean javadoc for complete details.
2018-03-28 17:21:25.889  WARN 9912 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.jobScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean javadoc for complete details.
2018-03-28 17:21:25.897  INFO 9912 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Replacing scope 'step' from [org.springframework.batch.core.scope.StepScope@388ba540] to [org.springframework.batch.core.scope.StepScope@47605f2f]
2018-03-28 17:21:25.899  INFO 9912 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'itemReader' with a different definition: replacing [Generic bean: class [com.example.demo.step.Reader]; scope=step; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=null; factoryMethodName=reader; initMethodName=null; destroyMethodName=null; defined in class path resource [batchjob.xml]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in class path resource [batchjob.xml]]
2018-03-28 17:21:25.899  INFO 9912 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'itemProcessor' with a different definition: replacing [Generic bean: class [com.example.demo.step.Processor]; scope=step; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in class path resource [batchjob.xml]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in class path resource [batchjob.xml]]
2018-03-28 17:21:25.899  INFO 9912 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'itemWriter' with a different definition: replacing [Generic bean: class [com.example.demo.step.Writer]; scope=step; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in class path resource [batchjob.xml]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in class path resource [batchjob.xml]]
2018-03-28 17:21:26.210  INFO 9912 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2018-03-28 17:21:26.225  INFO 9912 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2018-03-28 17:21:26.225  INFO 9912 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.28
2018-03-28 17:21:26.229  INFO 9912 --- [ost-startStop-1] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\Program Files\Java\jre1.8.0_161\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:/Program Files/Java/jre1.8.0_161/bin/server;C:/Program Files/Java/jre1.8.0_161/bin;C:/Program Files/Java/jre1.8.0_161/lib/amd64;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\AMD\ATI.ACE\Core-Static;C:\Program Files\PuTTY\;C:\Program Files\Git\cmd;C:\Users\XXXel\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\maven3.2.5\apache-maven-3.2.5\bin;;C:\Program Files (x86)\eclipse\eclipse;;.]
2018-03-28 17:21:26.285  INFO 9912 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2018-03-28 17:21:26.286  INFO 9912 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 1196 ms
2018-03-28 17:21:26.397  INFO 9912 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Servlet dispatcherServlet mapped to [/]
2018-03-28 17:21:26.417  INFO 9912 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
2018-03-28 17:21:26.417  INFO 9912 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2018-03-28 17:21:26.417  INFO 9912 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2018-03-28 17:21:26.417  INFO 9912 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
2018-03-28 17:21:26.478  INFO 9912 --- [           main] o.s.j.d.DriverManagerDataSource          : Loaded JDBC driver: oracle.jdbc.OracleDriver
2018-03-28 17:21:26.942  INFO 9912 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5af97850: startup date [Wed Mar 28 17:21:25 CEST 2018]; root of context hierarchy
2018-03-28 17:21:27.005  INFO 9912 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/runjob]}" onto public java.lang.String com.example.demo.controller.WebController.handle() throws java.lang.Exception
2018-03-28 17:21:27.009  INFO 9912 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2018-03-28 17:21:27.010  INFO 9912 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2018-03-28 17:21:27.038  INFO 9912 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-03-28 17:21:27.038  INFO 9912 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-03-28 17:21:27.069  INFO 9912 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-03-28 17:21:27.396  INFO 9912 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: ORACLE
2018-03-28 17:21:27.409  INFO 9912 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2018-03-28 17:21:27.595  INFO 9912 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2018-03-28 17:21:27.633  INFO 9912 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2018-03-28 17:21:27.635  INFO 9912 --- [           main] c.e.demo.SpringMultiprocessApplication   : Started SpringMultiprocessApplication in 2.845 seconds (JVM running for 3.316)
2018-03-28 17:21:27.638  INFO 9912 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: []
2018-03-28 17:21:27.745  INFO 9912 --- [           main] o.s.b.f.xml.XmlBeanDefinitionReader      : Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
2018-03-28 17:21:27.771  INFO 9912 --- [           main] o.s.jdbc.support.SQLErrorCodesFactory    : SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase, Hana]
2018-03-28 17:21:27.809  INFO 9912 --- [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2018-03-28 17:21:27.816 ERROR 9912 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Failed to execute CommandLineRunner
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:793) [spring-boot-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774) [spring-boot-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246) [spring-boot-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1234) [spring-boot-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at com.example.demo.SpringMultiprocessApplication.main(SpringMultiprocessApplication.java:18) [classes/:na]
Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = ? order by JOB_INSTANCE_ID desc]; nested exception is java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

    at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:235) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1402) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:620) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:657) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:688) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:700) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.getJobInstances(JdbcJobInstanceDao.java:230) ~[spring-batch-core-4.0.0.RELEASE.jar:4.0.0.RELEASE]
    at org.springframework.batch.core.explore.support.SimpleJobExplorer.getJobInstances(SimpleJobExplorer.java:173) ~[spring-batch-core-4.0.0.RELEASE.jar:4.0.0.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161]
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_161]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_161]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_161]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) ~[spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) ~[spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) ~[spring-batch-core-4.0.0.RELEASE.jar:4.0.0.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at com.sun.proxy.$Proxy80.getJobInstances(Unknown Source) ~[na:na]
    at org.springframework.batch.core.JobParametersBuilder.getNextJobParameters(JobParametersBuilder.java:254) ~[spring-batch-core-4.0.0.RELEASE.jar:4.0.0.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:162) ~[spring-boot-autoconfigure-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) ~[spring-boot-autoconfigure-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) ~[spring-boot-autoconfigure-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) ~[spring-boot-autoconfigure-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:790) [spring-boot-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    ... 5 common frames omitted
Caused by: java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
    at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:666) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:605) ~[spring-jdbc-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    ... 27 common frames omitted

2018-03-28 17:21:27.817  INFO 9912 --- [           main] ConfigServletWebServerApplicationContext : Closing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5af97850: startup date [Wed Mar 28 17:21:25 CEST 2018]; root of context hierarchy
2018-03-28 17:21:27.818  INFO 9912 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown

I'm new to Spring, but what I understand is that Spring Batch is trying to access my database and query it for its metadata (SpringBatch stuff). Now the question is: how can I tell Spring Batch to use its embedded database to save its data (metadata) and not try to access my database?

application.properties

spring.batch.job.enabled=true
spring.batch.initialize-schema=EMBEDDED

spring.datasource.url=jdbc:oracle:thin:@localhost:1521:ASDF
spring.datasource.username=USERNAME
spring.datasource.password=PASSWORD
spring.datasource.driver-class-name=oracle.jdbc.OracleDriver

MultipleDBConfig.java Bean used to configure the database

@Configuration
public class MultipleDBConfig {

    @Autowired
    private Environment env;

    @Bean
    public DataSource customDataSource() {

        DriverManagerDataSource dataSource = new DriverManagerDataSource();

        dataSource.setDriverClassName(env.getProperty("spring.datasource.driver-class-name"));
        dataSource.setUrl(env.getProperty("spring.datasource.url"));
        dataSource.setUsername(env.getProperty("spring.datasource.username"));
        dataSource.setPassword(env.getProperty("spring.datasource.password"));

        return dataSource;
    }
}

Answer 1

This is how I've solved it in the past. This was non-boot and old Spring, but it might still work with whatever environment you're in. By default the setDataSource() will be injected with your Oracle ds, but if you prevent it from being injected, Spring Batch will just store those details in memory.

If you want a different datasource to be used (ie if you want the batch metadata to still be persisted), you could probably use @Qualifier to specify a different datasource bean and call super.setDataSource(dataSource); . Probably other ways to do it too, haven't touched this code in a few years.

@Configuration
@EnableBatchProcessing
public class BatchConfig extends DefaultBatchConfigurer {

    @Override
    @Autowired
    public void setDataSource(DataSource dataSource) {
        // If we don't provide a datasource, an in-memory map will be used.
    }
}