Parallel Processing using Java and Spring

Question

I have scenario where my Spring batch job is running every 3 mins.

Steps should be

1. Each user's record should get executed parallel. Each user can have maximum of 150k records.
2. Every user can have update and delete records. Update records should run before delete.
3. Update/delete sets should run parallel on their own. But strictly all updates should complete before delete.

Can anyone suggest the best approach to achieve the parallelism at multiple levels and follow the order at update and delete level. I am looking something around Spring Async Executor Service, Parallel Streams and other Spring libraries. Rx, only if it gives some glaring performance which the above specified can't provide.

Answer 1

Glaring performance is based on the design of spring batch implementation and we are sure you will get with spring batch as we are processing millions of records with select delete and update.

1. Each user's record should get executed parllely. Each user can have maximum of 1.5 lakh records. "Partition the selection based on User and Each user will run as parallel steps."
2. Every user can have update and delete records. Update records should run before delete. " Create a Composite Writer and delegates added for update Ist writer and delete 2nd writer "
3. Update/delete sets should run parallel on their own. But strictly all updates should complete before delete. "Each writer step update and delete manages the transaction and make sure update executes first ".

Please refer below

Spring Batch multiple process for heavy load with multiple thread under every process

Composite Writer Example

Spring Batch - Read a byte stream, process, write to 2 different csv files convert them to Input stream and store it to ECS and then write to Database