如何对单个事务多次调用@Transactional 方法

Question

I have a method

@Transactional
public void updateSharedStateByCommunity(List[]idList)

This method is called from the following REST API:

@RequestMapping(method = RequestMethod.POST)
public ret_type updateUser(param) {
  // call updateSharedStateByCommunity
}

Now the ID lists are very large, like 200000, When I try to process it, then it takes lots of time and on client side timeout error occurred.

So, I want to split it to two calls with list size of 100000 each.

But, the problem is, it is considered as 2 independent transactions.

NB: The 2 calls is an example, it can be divided to many times, if number ids are more larger.

I need to ensure two separate call to a single transaction. If any one of the 2 calls fails, then it should rollback to all operation.

Also, in the client side, we need to show progress dialog, so I can't use only timeout.

Answer 1

The most obvious direct answer to your question IMO is to slightly change the code:

@RequestMapping(method = RequestMethod.POST)
public ret_type updateUser(param) {
    updateSharedStateByCommunityBlocks(resolveIds);
}

...

And in Service introduce a new method (if you can't change the code of the service provide an intermediate class that you'll call from controller with the following functionality):

@Transactional
public updateSharedStatedByCommunityBlocks(resolveIds) {
    List<String> [] blocks = split(resolveIds, 100000);  // 100000 - bulk size
    for(List<String> block :blocks) {
       updateSharedStateByCommunity(block); 
    }
}

If this method is in the same service, the @Transactional in the original updateSharedStateByCommunity won't do anything so it will work. If you'll put this code into some other class, then it will work since the default propagation level of spring transaction is "Required"

So it addresses harsh requirements: you wanted to have a single transaction - you've got it. Now all the code runs in the same transaction. Each method now runs with 100000 and not with all the ids, everything is synchronous:)

However, this design is problematic for many different reasons.

1. It doesn't allow to track the progress (show it to the user) as you've stated by yourself in the last sentence of the question. REST is synchronous.

2. It assumes that network is reliable and waiting for 30 minutes is technically not a problem (leaving alone the UX and 'nervous' user that will have to wait:) )

3. In addition to that, the network equipment can force closing the connection (like load balancers with pre-configured request timeout).

That's why people suggest some kind of asyncrhonous flow.

I can say that you still can use the async flow, spawn the task, and after each bulk update some shared state (in-memory in the case of a single instance) and persistent (like database in the case of cluster).

So that the interaction with the client will change:

1. Client calls "updateUser" with 200000 ids
2. Service responds "immediately" with something like "I've got your request, here is a request Id, ping me once in a while to see what happens.
3. Service starts an async task and process the data chunk by chunk in a single transaction
4. Client calls "get" method with that id and server reads the progress from the shared state.
5. Once ready, the "Get" methods will respond "done".

If something fails during the transaction execution, the rollback is done, and the process updates the database status with "failure".

You can also use more modern technologies to notify the server (web sockets for example), but it's kind of out of scope for this question.

Another thing to consider here: from what I know, processing 200000 objects should be done in much less then 30 minutes, its not that much for modern RDBMSs. Of course, without knowing your use case its hard to tell what happens there, but maybe you can optimize the flow itself (using bulk operations, reducing the number of requests to db, caching and so forth).

Answer 2

My preferred approach in those scenarios is make the call asynchronous (Spring Boot allow this using the @Async annotation), hence the client won't expect for any HTTP response. The notification could be done via a WebSocket that will push a message to the client with the progress each X items processed.

Surely it will add more complexity to your application, but if you design the mechanism properly, you'll be able to reuse it for any other similar operation you may face in the future.

Answer 3

The @Transactional annotation accepts a timeout (although not all underlying implementations will support it). I would argue against trying to split the IDs into two calls, and instead try to fix the timeout (after all, what you really want is a single, all-or-nothing transaction). You can set timeouts for the whole application instead of on a per-method basis.

Answer 4

From technical point, it can be done with the org.springframework.transaction.annotation.Propagation#NESTED Propagation, The NESTED behavior makes nested Spring transactions to use the same physical transaction but sets savepoints between nested invocations so inner transactions may also rollback independently of outer transactions, or let them propagate. But the limitation is only works with org.springframework.jdbc.datasource.DataSourceTransactionManager datasource.

But for really large dataset, it still need more time to processing and make the client waiting, so from solution point of view, maybe using async approach will be more better but it depends on your requirement.