Spring Batch partitioning

Question

We are developing an application which need to read million records from a Table A and group it as sub groups (Table B) and master groups (Table C). We are using spring batch to do that. The problem is that grouping of records is based on the data in the Table A and there would be one master group for the each unique grouping criteria, and there would be one sub group for each 1000 record, if they fall into same master group.

So the structure looks like this.

Records,Table A-->Subgroup, Table B (for each 1000 records belongs to a unique grouping criteria)-->Master Group, Table C-->(for a unique grouping criteria)

If I do in a non partitioned step, concept is fine. But when I partition the step, how is it possible for individual partitions to know that the count has reached 1000, and a new subgroup need to be created. ?

Any better ideas to solve this problem is also appreciated.

Answer 1

I believe that partitioning needs to be separated batch job.

I would not go for anything complicated as 2-phase-commit or implementing custom registry table where you store a row for every partition id along with the count of rows that the partition contains.

Also you can use Spring-Batch ItemReaders-ItemWriters to implement some global trigger mechanism in java. It would store a map of partition IDs and counts and when some count reaches 1000 some global java task will be triggered. The advantage of this method over implementing something similar in DB is performance.