Questions about transaction management

Question

I'm trying to implement a business need with Spring and having some problems about transaction management.

The business need is, simply,
1) Query a database(#1) with some criteria.
2) Write the results of the query to a flat file.
3) Update those records in the database(#1) as "processed".
4) Insert records into another database (#2) as a result of the first 3 steps.

I need those 4 steps to be trasanctional. For example, if the 4th step fails, the last item written to the flat file in the 2nd step should be rolled back. I found "Apache Commons File Resource Manager" to implement rollback mechanism with files.

My mentor recommended me to use Spring Batch generally for these kind of projects. However, considering the fact that Spring Batch uses the "chunk oriented processing" style, it looks quite difficult to implement it with Spring Batch. Because, as far as I know, chunk oriented style requires the 2nd step to be completed for the whole chunk and then it allows to move on to the 3rd and 4th step in my implementation.

I think I need some general advice about this project and whether or not to use Spring Batch for this purpose. Furthermore, is it a good idea to try to implement these needs in a concurrent environment since the rollback mechanism will be quite complex for that style.

Answer 1

The purpose of batch processing is to break down what you are trying to do into small units of work that can be processed individually (sequentially or in parallel). Spring batch is a framework that provides you with a flexible mechanism built on top of Spring that allows you to perform batch style processing. Based on your requirements, you can configure Spring batch to execute in many different ways. The chunk oriented processing you mentioned is but one way that Spring batch can be used.

If I look at the information you provided, here are the elements of your application as I see them:

• read records from database that need to be processed
• process the records, which results in a db #2 being updated
• mark records as processed
• write record to processed file

You will notice that I rearranged the order of events a little. The logic as I described is something well suited for batch-style processing. Here is an example of how you could manage this processing with Spring batch:

1. Retrieve records from database to be processed. The retrieval would be done through a custom database retriever. These records will be used as a record selector by Spring Batch. You would then configure Spring batch to iterate over this list.
2. You would configure the next step of Spring batch to process each of the records retrieved in step 1. This processing would consist of processing an individual record and writing the results to the second db.
3. Your final step would be to conclude the processing of an individual record. As with most Spring frameworks, there are many different ways you can configure when this occurs. This final step would perform two functions: update record in db #1 as processed and write the processed record to the output file. The writing of the output file could be performed as a step 4 on completion of the overall batch process, where you would look at the records marked as processed during the current run and write the data as necessary.

Spring batch does let you configure how transactions are managed throughout its processing, so you could configure a transaction that goes across all the actions and if the file write fails, then it can be configured to rollback the rest of the database operations.

With this approach, you could process the records sequentially, in parallel, etc.. However, all of this assumes that you can break up the data into smaller units of work. If that is not the case, then batch processing is not really something that can be performed as you only have one process.

I'd like to say there is an easy answer to decide whether to use Spring batch or not. It is a platform that lets you do a lot of things. Along with that flexibility it does have a relatively high learning curve.

Hope this helps.

Answer 2

I'd say the Spring way to do it is to implement a Flat File version of Spring's PlatformTransactionManager abstraction that would hold back the file write until the transaction is committed. Shouldn't be too hard.