Spring Batch : Reader Issue : Multiple input queries for forming domain object

Question

I have a reader query which fetches product ids from database.

My domain object contains several lists as well.

My commit count is 1000.

So, my sql fetches 1000 items (ie 1000 product ids) at a time.

However, in my rowmapper, I have 10 other queries which I run for each Product Id fetched in the chunk. These queries returns me a number of rows as result set. I parse these result sets to set the lists in my domain object.

This is the way one complete domain object gets created, which I then pass on to the writer for writing into a XML using suitable mapper.

However, since these queries in rowmapper run for every record fetch, my overall program performance is getting hit severely.

There is no way I can combine these queries with the main reader query.

Is there any solution for such a requirement to improve performance ? I there any way I can do some parallel processing with respect to the queries running in rowmapper to reduce the processing time ?

Thanks for reading!

Answer 1

To your specific questions:

1. Is there any solution for such a requirement to improve performance - Yes. There could be many solutions depending on your particular use case (database design, are any of the queries cacheable, etc).
2. I there any way I can do some parallel processing with respect to the queries running in rowmapper to reduce the processing time - With this, I have to ask if there is the ability to have the reader return just the domain object with the id populated and have an ItemProcessor do the rest of the queries. I ask because there is a project called Spring Batch Integration ( https://github.com/spring-projects/spring-batch-admin/tree/master/spring-batch-integration ) that provides an AsyncItemProcessor. You could use this to have each item processed in parallel (doing the look ups) then the corresponding AsyncItemWriter would write out the result once the lookups are complete. This would be the quickest and easiest way to accomplish parallelizing this type of logic.

Other options exist for parallelizing this type of scenario, however do keep in mind, running many little queries will beat up your database. Running them in parallel will only compound that problem. You may need to reconsider how those sub queries are structured.