Best practice for when to start a new session/transaction for batch jobs using spring/hibernate and when to commit/flush the session?

Question

There's a tx-advice i have set in spring to wrap a transaction around my Service methods. So say in my batch class, i call a service method to load a list of objects and return it to my batch class. Then in my batch class i call a service method to process each of those objects. But i'll receive a lazy loading exception if that service method tries to access a lazy loaded property of the object since that list of objects was loaded with a different hibernate session.

So a way around this which may not be the most optimal is - the batch class just calls a service to load all the IDs of those objects (long values) - and we pass this ID to a service method which will load that object from the DB by the ID and then do the processing on it.

Thoughts on this?

Another question I had was if each of these objects are independent of each other, should or should i not persist each object one at a time vs persisting them all at once or batch it. If theres 5000 records, it seems like the application slows down a lot when calling save/update/insert since it's still doing all that in memory in the Hibernate Session. But if I instead save/update/insert each record (processing one ID at a time) and then committing when done w/ that object before going to the next, it seems to speed up a lot. Also if i batch it, say every 200 or even do all 5000 at once, if one record fails to insert/update and gets an error, nothing will get persisted and everything rolls back.

What are the best practices for handling things like this? Seems like something really common. Thanks

Answer 1

First of all, Spring/Hibernate are not really intended for batch processing. Instead, check out either Talend or Pentaho (if you are into open source), or any of a huge (massive!) variety of commercial tools. Either of these tools can be used to automatically generate a lump of Java code that will do exactly what you need (including insert optimization, elegant error handling, etc).

Ok, let's assume that you really, really want to make Spring/Hibernate do batch processing. You have a couple of different issues - first, the Hibernate session lifecycle means that objects loaded expect to be associated to the live session. You can use the session flush() to force the changes to propagate to the database. Session close() will wipe everything out. Objects that are already loaded can only be reattached to a new session with difficulty (usually it's easier to just reload the object). If you don't close()/flush() your session, eventually you will (probably) run out of memory. You can fix that by adding a Hibernate 2nd level cache... but that will just make things more complex and slow it down.

There is no real reason not to just do each insertion within an independent Hibernate session (open, do work, close). It won't be as fast as a dedicated tool, but it's simple, will work fine, and is more or less as good as you'll get.

Answer 2

with regards to the batching requirment, please use Spring batching link this provides all the necessary batching facilities needed.

Regarding the object loading issue,

So a way around this which may not be the most optimal is - the batch class just calls a service to load all the IDs of those objects (long values) - and we pass this ID to a service method which will load that object from the DB by the ID and then do the processing on it.

seems correct.