Spring Batch - Remote Partitioning in Manager - Worker Environment - CSV Files

Question

I have a requirement where i need to work with 400+ CSV files (each of 100 to 500MB) which were stored in a directory / GCP.

For each file i need to get valid account numbers from DB and keep them in memory, validate each row of file against data in memory and write the valid records in to another CSV file (column format changes) and export generated files to AWS. In future, would like to support requests coming from MQ and process remain same.

Requirement is to implement this in distributed system.

I am planning to use Spring Batch for this requirement, But i am unable to find good example / code of Remote partitioning using different systems as Manager - Worker. I want to understand how to configure Manager, Worker, how to start them and pass file metadata as request and response.

I got examples of partitioning on single JVM, so looking for Manager-Worker example only with sample code.

Any suggestions / references will be highly appreciated and Thanks in advance.

Answer 1

I am planning to use Spring Batch for this requirement, But i am unable to find good example / code of Remote partitioning using different systems as Manager - Worker.

Remote partitioning is explained in details with code examples and diagrams here: https://docs.spring.io/spring-batch/docs/4.2.x/reference/html/spring-batch-integration.html#remote-partitioning

You can also find:

• a complete sample here: Remote partitioning sample along with the code for both Manager and Worker setup: https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples/src/main/java/org/springframework/batch/sample/remotepartitioning
• a talk about scaling options in Spring Batch here: https://youtu.be/J6IPlfm7N6w?t=2230