Java Batch job in cluster environment

Question

We have a cluster with 2 JBOSS nodes. We have a batch job which loads all users details from a active directory to a DB. This job is run everyday. It was run before in a non clustered environment and hence we designed it as a singleton. Now we have a clustered environment and I do not know what is best way to achieve the same result. I want batch job to be run only once a day. We use spring and hibernate and I looked at Spring batch. I could not get any concise answer to my question.

Can anybody please let me know if you had implemented batch in cluster environment? What would be the best solution in this scenario?

Answer 1

We implemented this by triggering and starting the jobs externally via MQ ( an http request to start the job would work as well). The scheduler puts a message on the queue and even though we have 'n' nodes listening to the queue, one node will receive the message and based on it's contents, start the job. You can do this with HTTP as well.

The real 'solution' to this is to schedule the batch job 'externally' and not via an internal cron trigger. The actual start mechanism is secondary to that.

Answer 2

Consider also https://github.com/willschipp/spring-batch-cluster that features

• write-behind for the Batch Job Repository
• HA for batch in a cluster (automatic stop and failover of executing jobs)

Answer 3

In general, it's sometimes a good idea to externalize/isolate batch jobs from transactional systems, so they don't interfere with availability or performance. That being said, if there are good reasons for embedding a batch job in a clustered application (simplicity, code reuse, etc.), then aside from solutions already mentioned, ShedLock is a great option.

Note for whatever its worth, a blog post about batch jobs in clustered environments .