Getting database deadlock with @Transactional in spring boot and hibernate

Question

Why I am getting deadlock in this code?

I tried to debug it and also read many article about deadlock prevention but could not get this. I have used synchronization, to make thread safe a block of code on the basis of accountNumber .

I am getting this Transaction object from an API and I want to lock my code on the basis of what the Transaction object contain. Transaction object contains info like debit/credit account number, amount etc.

Two threads should not be executed executeTransaction method simultaneously if there is any common accountNumber between them.

Here, lockedAccount is storing all accounts that are currently locked and two methods for locking and unlocking an accountNumber .

DAO / Repository layer.

@Repository
public class TransactionDAOImpl implements TransactionDAO {

    // define field for entitymanager
    private EntityManager entityManager;

    public TransactionDAOImpl() {}
    // set up constructor injection
    @Autowired
    public TransactionDAOImpl(EntityManager theEntityManager) {
        entityManager = theEntityManager;
    }


    private static final Set<String> lockedAccounts = new HashSet<>();

    private void LockAccount(String AccountNumber) throws InterruptedException {
        int count = 0;
        synchronized (lockedAccounts) {
            while (!lockedAccounts.add(AccountNumber)) {
                lockedAccounts.wait();
                count++;
            }
            System.out.println(AccountNumber + " waited for " + count + " times" + " and now i am getting lock");
        }
    }

    private void unLockAccount(String AccountNumber) {
        synchronized (lockedAccounts) {
            lockedAccounts.remove(AccountNumber);
            lockedAccounts.notifyAll();
            System.out.println("unlocking " + AccountNumber);
        }
    }

    
    @Override
    public void executeTransaction(Transaction theTransaction) {
        // System.out.println(theTransaction);
        // get the current hibernate session
        Session currentSession = entityManager.unwrap(Session.class);

        // lock both account in a increasing order to avoid deadlock
        // lexicographically lesser account number should be lock first
        String firstAccount = theTransaction.getDebitAccountNumber();
        String secondAccount = theTransaction.getCreditAccountNumber();
        // check firstAccount is lesser or greater then second account,if not then swap
        // value
        if (firstAccount.compareTo(secondAccount) > 0) {
            firstAccount = theTransaction.getCreditAccountNumber();
            secondAccount = theTransaction.getDebitAccountNumber();
        }
        try {
            LockAccount(firstAccount);
            try {
                LockAccount(secondAccount);
               
                AccountDetail debitAccount = getAccountDetails(currentSession, theTransaction.getDebitAccountNumber());
                AccountDetail creditAccount = getAccountDetails(currentSession,
                        theTransaction.getCreditAccountNumber());

                if (debitAccount == null || creditAccount == null) // check invalid accountNumber
                {
                    theTransaction.setStatus("failed,account not found");
                } else if (debitAccount.getBalance() < theTransaction.getAmount()) // check insufficient account balance
                {
                    theTransaction.setStatus("failed,insufficient account balance");
                } else {
                    // update custmer accout balance
                    debitAccount.setBalance(debitAccount.getBalance() - theTransaction.getAmount());
                    creditAccount.setBalance(creditAccount.getBalance() + theTransaction.getAmount());
                   
                    // save to database
                    currentSession.saveOrUpdate(debitAccount);
                    currentSession.saveOrUpdate(creditAccount);

                    // update status of transacion
                    theTransaction.setStatus("successful");
                }
            } catch (InterruptedException e) {
                e.printStackTrace();
            } finally {
                unLockAccount(secondAccount);
            }
        } catch (InterruptedException e1) {
            e1.printStackTrace();
        } finally {
            unLockAccount(firstAccount);
        }
        return;
    }
     private AccountDetail getAccountDetails(Session currentSession, String accountNumber) {

        Query<?> query = currentSession.createQuery("from AccountDetail where accountNumber=:accountNumber");
        query.setParameter("accountNumber", accountNumber);
        AccountDetail accountDetails = (AccountDetail) query.uniqueResult();
        return accountDetails;
    }
}
    

for more information, my accountDetails table in database have three columns,

id(int,primary key)

AccountNumber(String,unique)

amount(double)

this is Service layer where i am using @Transactional annotation for executeTransaction method.

public class TransactionServiceImpl implements TransactionService {

    private TransactionDAO theTransactionDAO;
    
    public TransactionServiceImpl() {}
    
    //constructor injection
    @Autowired
    public TransactionServiceImpl(TransactionDAO theTransactionDAO)
    {
        this.theTransactionDAO= theTransactionDAO;
    }
    
    @Override
    @Transactional
    public void executeTransaction(Transaction theTransaction) {
        theTransactionDAO.executeTransaction(theTransaction);
    }
}

but i am getting database deadlock in this code. below is my error.

2020-08-30 19:09:28.235  WARN 6948 --- [nio-8081-exec-4] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 1213, SQLState: 40001
2020-08-30 19:09:28.236 ERROR 6948 --- [nio-8081-exec-4] o.h.engine.jdbc.spi.SqlExceptionHelper   : Deadlock found when trying to get lock; try restarting transaction
2020-08-30 19:09:28.384 ERROR 6948 --- [nio-8081-exec-4] o.a.c.c.C.[.[.[.[dispatcherServlet]      : Servlet.service() for servlet [dispatcherServlet] in context with path [/bank] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement] with root cause

com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:123) ~[mysql-connector-java-8.0.21.jar:8.0.21]
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.21.jar:8.0.21]
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.21.jar:8.0.21]
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) ~[mysql-connector-java-8.0.21.jar:8.0.21]
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092) 

Answer 1

Suppose there are two Account Transactions( debitAccount , creditAccount ): AT1 (1,2) AT2 (2,1). And we have Java Lock ( JL ) and Database Lock ( DBL ). In following scenario, deadlock will occur.

+------+---------------------+---------------------+-----------------------------------------------------+  
| Step | AT1 State           | AT2 State           | Remark                                              |  
+------+---------------------+---------------------+-----------------------------------------------------+  
| 1    | get JL              | wait JL             |                                                     |  
+------+---------------------+---------------------+-----------------------------------------------------+  
| 2    | release JL          | get JL              | AT1 saveOrUpdate may not flush to database,         |  
|      |                     |                     | hence database lock may not be acquired this moment |  
+------+---------------------+---------------------+-----------------------------------------------------+  
| 3    | flush debitAccount  | flush debitAccout   | AT1 acquire DB lock for account 1,                  |  
|      | saveOrUpdate        | saveOrUpdate        | AT2 acquire DB lock for account 2                   |  
+------+---------------------+---------------------+-----------------------------------------------------+  
| 4    | AT1 DBL account 1   | AT2 DBL account 2   |                                                     |  
+------+---------------------+---------------------+-----------------------------------------------------+  
| 5    | flush creditAccount | flush creditAccount | AT1 acquire DBL for account 2,                      |  
|      | saveOrUpdate        | saveOrUpdate        | AT2 acquire DBL for account 1, Deadlock             |  
+------+---------------------+---------------------+-----------------------------------------------------+  

Please also note that

1. Database lock is acquired in update statement when the statement is flushed.
2. Database lock is released when transaction commit/rollback.