Spring batch remote partitioning to multiple application instances

Question

I am working on remote partitioning using spring batch.. I have multiple instances of spring batch application running. As part of use case I need to process all records present in database. So I am assigning instances to every records.. So each instance of batch application can process instance specific records.

eg

Records1 -: instance1
Records2 -: instance1
Records3 -: instance2
Records4 -: instance2
.... so on

I know we can use Kafka or JMS instead of partitioning explicitly. In my use case I don't want to use any messaging middleware. So after assigning instances to each records. I want to invoke both instances. How can I invoke both instances?

Answer 1

You still need a messaging middleware to implement remote partitioning. This is required for the manager step to send StepExecutionRequest s to worker steps.

If you really want to avoid having a messaging middleware, you can create a job instance per record set. According to your example, this could be designed as follows:

• Job instance 1 -> Records1 and Records2
• Job instance 2 -> Records3 and Records4
• etc

Those job instances can be executed locally in different threads of the same JVM, or locally in different JVMs on the same machine, or remotely on different machines in a cluster. The degree of parallelism depends only on the resources available in your cluster. You can have, on a single machine, several JVMs where each JVM is running several job instances in parallel. You could then duplicate that setup on different machines and achieve a high level of parallelism for you requirement.