Prevent spring batch from crashing

Question

I am currently using spring-batch, and there has been an error on our test servers for several month. We don't have direct access to these server, and I can't reproduce this error in our environment, all I have is this stacktrace (see after).

My main problem is, this exception stops the whole batch. Is there a work-around ? some way to catch it and continue ? The problem does not directly come from one of our reader/processer/writer, it's between them, and I don't know where, I'm guessing during the write, but there is this "at java.net.SocketInputStream.read(SocketInputStream.java:168)" ...

Here is the full stacktrace (more informations after this):

    14/04/2016 15:18:46.322 [ERROR] myApp - org.springframework.batch.core.step.AbstractStep            Encountered an error executing step myStep1 in job myJob
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:271) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:77) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:368) ~[spring-batch-infrastructure-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144) ~[spring-batch-infrastructure-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:198) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:165) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:362) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:590) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
    at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:170) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    ... 21 common frames omitted
Caused by: org.hibernate.exception.JDBCConnectionException: Could not open connection
    at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:235) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1435) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
    ... 23 common frames omitted
Caused by: org.postgresql.util.PSQLException: La tentative de connexion a échoué. (Note: "The connection attempt failed" in english)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:225) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.Driver.makeConnection(Driver.java:393) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.Driver.connect(Driver.java:267) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at java.sql.DriverManager.getConnection(DriverManager.java:582) ~[na:1.6.0_45]
    at java.sql.DriverManager.getConnection(DriverManager.java:154) ~[na:1.6.0_45]
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:153) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:144) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:155) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:120) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
    ... 28 common frames omitted
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[na:1.6.0_45]
    at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:376) ~[postgresql-9.2-1002.jdbc4.jar:na]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) ~[postgresql-9.2-1002.jdbc4.jar:na]
    ... 45 common frames omitted

As notes and side questions:

When i first wrote the post, the question was "I want to know when/where in the job this exception is thrown ?". I could use this to prevent the crash, but actually, any solution is good.

Other datas: The stacktrace is the main clue I have. The spring batch tables indicates that 79000 items have been read ( pagesize is 50000 for the reader but commit-interval is 500) until it crashes. So I don't know if the tables are up to date when evrything stops.

Yes, I'm using DriverManagerDataSource which is not a viable pool manager. I'm not sure a real connection pool manager would change anything (like, retry on fail or whatever ? does that exist ? would it work here ?), and most informations on DBCP and c3p0 seems outdated (any recommandation if one could prevent the error would be great). 99.9% of the data throws an exception in the processor for business/flow control (I know, but can't change) and there are some skippable-exception-classes , but they are custom exceptions.

For the exception that causes the whole thing to crash ( org.postgresql.util.PSQLException and java.net.SocketException: Connection reset ) I can't reproduce it, neither see why the connection is reset: I've seen some thread about this error, but I can't access the postgresql logs, and some people said that this error might even be due to a proxy that sets a timeout on its own (or something like that). I've even looked at the source code of the classes from the last stack trace displayed, but it didn't help. So I don't think I can do anything about this error.

Thank you.

Answer 1

JDBCConnectionException: Connection refused or Connection timed out which means Database is either down or rejecting connection.

First of all check the data sources is being used in your applications, including in all config files is used in hibernate xml's. There might be possibility the database may not be reachable. Please check the IP address or host name, Port, DB may not accept TCP/IP connections, firewall or proxy.