Spring batch integration remote partitioning - running parallel jobs

Question

We have a usecase where we need to read data from some paginated API and write to some downstream Kafka topic.

We have been able to implement the solution via spring batch integration remote partitioning where the manager takes care of partitioning the task by creating executionContext which container the page number and offset to read data for. Manager creates this executionContext and puts them on the messagingChannel(I can use rabbitMQ and Kafka topic whichever provides a solution). The workers(more than 1) picks those executionContext from he messagingChannel and completes the task of reading the data from the API and writing it to required Kafka topic.

The above implementation works just fine. This also works fine if I run the same job for different clients one after another. The challenge comes when we want to run the same job for multiple clients in parallel . For example, we launch the jobs for 2 client in parallel. It creates 1 manager and 2 workers for each client. Now the issue comes when both of the manager pushed the executionContext on the same messagingChannel and workers don't know which one to pick and execute. Also, both the jobs share the same database spring batch tables , so I suspect it would create problems at that level as well.

Any input or references on how to implement running multiple spring batch reporter partitioning jobs in parallel

Answer 1

In such a setup, the MessageChannelPartitionHandler should be step-scoped. There is a note about that in the Javadoc:

Note: The reply channel for this is instance based.
Sharing this component across multiple step instances may result in the
crossing of messages. It's recommended that this component be step or job scoped.

Making this bean step-scoped should fix the issue.