Java / EJB / Hibernate事务同步以防止Oracle死锁

Question

I've got a Quartz job that runs every 5 minutes and I'm seeing deadlock errors in the application.log and Oracle trace logs. I'm looking to resolve these deadlock errors, but I'm not sure if they're happening due to the locking strategy, transaction isolation, or a Quartz concurrency-related issue. Please let me know what might be wrong with my locking strategy, what I might be missing, or otherwise what steps I should take to resolve this.

I'm creating the TypedQuery which will lock the row(s) like so:

TypedQuery<Order> ordersQuery = em.createQuery("SELECT o from Order o where o.orderDate <= :today and o.statusId = :statusId", Order.class);

and setting the LockModeType with:

ordersQuery.setLockMode(LockModeType.PESSIMISTIC_WRITE);

The Quartz job calls the following methods in a stateless session bean . It first gets the orderIds to process (which is where the "for update" locking happens), then calls initiateOrders(List) with those orderIds. There are no TransactionAttributeTypes specified on either the class or these methods, so it should be defaulting to TransactionAttributeType.REQUIRED.

public int initiateOrders(List<Integer> orderIds) throws Exception {
    if (orderIds.size() == 0) {
      return 0;
    }

    List<Order> orders = orderDao.findByIds(orderIds);

    for (Order o : orders) {
      o.setProcessId(-1);
      boolean isSuccess;

      if (o.getOrderSummaryRequiredBy() != null && o.getOrderSummaryRequiredBy().equals("ACC")) {
        o.setRouteToDept(o.getRouteToEntity());
      }

      isSuccess = orderManager.startOrderProcess(o);
      invokeUpdateOrderTemplateActiveYN(o.getOrderId());
      if (!isSuccess) {
        log.error("Exception in startOrderProcess( " + o.getOrderId() + " )");
      }
    }

    return orders.size();
}

public List<Integer> getOrderIdsToProcess(int packetSize) {
    TypedQuery<Order> ordersQuery = entityManager
        .createQuery(
            "SELECT o from Order o where o.orderDate <= :today and o.statusId = :statusId and o.processId = :processId",
            Order.class);

    java.sql.Date today = new java.sql.Date(new Date().getTime());

    ordersQuery.setParameter("today", today);
    ordersQuery.setParameter("statusId", 1);
    ordersQuery.setParameter("processId", 0);
    ordersQuery.setMaxResults(packetSize);

    ordersQuery.setLockMode(LockModeType.PESSIMISTIC_WRITE);

    List<Order> orders = ordersQuery.getResultList();
    List<Integer> orderIds = new ArrayList<>();

    for (Order o : orders) {
      o.setProcessId(-1);
      orderIds.add(o.getOrderId());
    }

    return orderIds;
}

startOrderProcess(Order) is also in a stateless bean with a default transaction attribute type. It sets some values on the order object and calls entityManager.merge(order);

invokeUpdateOrderTemplateActiveYN(Integer) calls a stored procedure which selects from Order and updates a different table.

Application log (thinned down a bit and PII removed):

15:31:42,969 DEBUG [org.hibernate.SQL] (myScheduler_Worker-10) select MY_SEQ.nextval from dual
15:31:42,979 DEBUG [org.hibernate.SQL] (myScheduler_Worker-10) select assignedus0_.ORDER_ID as ORDER_2_90_1_, from ASSIGNED_USERS assignedus0_ where assignedus0_.ORDER_ID=?
15:31:42,979 DEBUG [org.hibernate.SQL] (myScheduler_Worker-10) insert into ORDER_EVENT_LOG (ACTION, ... EVENT_LOG_ID) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
15:31:42,979 DEBUG [org.hibernate.SQL] (myScheduler_Worker-10) update ORDER set ... where ORDER_ID=?
15:31:47,513 DEBUG [org.hibernate.SQL] (myScheduler_Worker-5) select user0_.USER_ID as USER_ID1_116_0_, user0_.ACTIVE_YN as ACTIVE_Y2_116_0_, ... from USER user0_ where user0_.USER_ID=?
15:31:47,533 DEBUG [org.hibernate.SQL] (myScheduler_Worker-5) select orderinf0_.ORDER_INFORMATION_ID as ORDER_1_88_0_, ... from ORDER_INFORMATION orderinf0_ where orderinf0_.ORDER_INFORMATION_ID=?
15:31:47,673 DEBUG [org.hibernate.SQL] (myScheduler_Worker-5) select orderinf0_.ORDER_INFO_CATEGORY_ID as ORDER_1_89_0_, ... from ORDER_INFORMATION_CATEGORY orderinf0_ where orderinf0_.ORDER_INFO_CATEGORY_ID=?
15:31:47,673 DEBUG [org.hibernate.SQL] (myScheduler_Worker-5) select user0_.USER_ID as USER_ID1_116_0_, user0_.ACTIVE_YN as ACTIVE_Y2_116_0_,... where user0_.USER_ID=?
15:31:48,151 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (myScheduler_Worker-2) SQL Error: 60, SQLState: 61000
15:31:48,151 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (myScheduler_Worker-2) ORA-00060: deadlock detected while waiting for resource

15:31:48,181 DEBUG [org.hibernate.SQL] (myScheduler_Worker-10) select orderdis0_.ORDER_ID as ORDER_4_90_1_, ... from ORDER_DISPATCH orderdis0_ where orderdis0_.ORDER_ID=?
15:31:48,181 ERROR [org.jboss.as.ejb3.invocation] (myScheduler_Worker-2) JBAS014134: EJB Invocation failed on component OrderManager for method public java.util.List com.business.model.order.OrderManager.getOrdersToProcess(int): javax.ejb.EJBException: javax.persistence.PessimisticLockException: could not obtain pessimistic lock
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:191) [jboss-as-ejb3-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
    ...
    at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
    at com.business.model.order.OrderManager$$$view26.getOrdersToProcess(Unknown Source) [model-1.0.jar:]
    at com.business.workflow.jobs.OrderProcessor.execute(OrderProcessor.java:59) [workflow-jobs-1.0.jar:]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-1.6.0.jar:1.6.0]
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529) [quartz-1.6.0.jar:1.6.0]
Caused by: javax.persistence.PessimisticLockException: could not obtain pessimistic lock
    at org.hibernate.ejb.AbstractEntityManagerImpl.wrapLockException(AbstractEntityManagerImpl.java:1449) [hibernate-entitymanager-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ...
    at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:277) [hibernate-entitymanager-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    at com.business.model.order.OrderManager.getOrdersToProcess(OrderManager.java:2655) [model-1.0.jar:]
    ... 27 more
Caused by: org.hibernate.dialect.lock.PessimisticEntityLockException: could not obtain pessimistic lock
    at org.hibernate.dialect.lock.PessimisticWriteSelectLockingStrategy.lock(PessimisticWriteSelectLockingStrategy.java:114) [hibernate-core-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ... 59 more
Caused by: org.hibernate.exception.LockAcquisitionException: could not extract ResultSet
    ...
    at org.hibernate.dialect.lock.PessimisticWriteSelectLockingStrategy.lock(PessimisticWriteSelectLockingStrategy.java:87) [hibernate-core-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ... 81 more
Caused by: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450) [ojdbc-7.jar:12.1.0.1.0]
    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399) [ojdbc-7.jar:12.1.0.1.0]
    ...
    at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)
    at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) [hibernate-core-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ... 82 more

15:31:48,191 INFO  [org.quartz.core.JobRunShell] (myScheduler_Worker-2) Job workflow.OrderProcessor threw a JobExecutionException: : org.quartz.JobExecutionException: javax.ejb.EJBException: javax.persistence.PessimisticLockException: could not obtain pessimistic lock [See nested exception: javax.ejb.EJBException: javax.persistence.PessimisticLockException: could not obtain pessimistic lock]
    at com.business.workflow.jobs.OrderProcessor.execute(OrderProcessor.java:66) [workflow-jobs-1.0.jar:]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-1.6.0.jar:1.6.0]
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529) [quartz-1.6.0.jar:1.6.0]
Caused by: javax.ejb.EJBException: javax.persistence.PessimisticLockException: could not obtain pessimistic lock
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:191) [jboss-as-ejb3-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
    ...
    at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
    at com.business.model.order.OrderManager$$$view26.getOrdersToProcess(Unknown Source) [model-1.0.jar:]
    at com.business.workflow.jobs.OrderProcessor.execute(OrderProcessor.java:59) [workflow-jobs-1.0.jar:]
    ... 2 more
Caused by: javax.persistence.PessimisticLockException: could not obtain pessimistic lock
    at org.hibernate.ejb.AbstractEntityManagerImpl.wrapLockException(AbstractEntityManagerImpl.java:1449) [hibernate-entitymanager-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ...
    at com.business.model.order.OrderManager.getOrdersToProcess(OrderManager.java:2655) [model-1.0.jar:]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
    ...
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:280) [jboss-as-ejb3-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
    ... 27 more
Caused by: org.hibernate.dialect.lock.PessimisticEntityLockException: could not obtain pessimistic lock
    at org.hibernate.dialect.lock.PessimisticWriteSelectLockingStrategy.lock(PessimisticWriteSelectLockingStrategy.java:114) [hibernate-core-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ...
    at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268) [hibernate-entitymanager-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ... 59 more
Caused by: org.hibernate.exception.LockAcquisitionException: could not extract ResultSet
    at org.hibernate.dialect.Oracle8iDialect$2.convert(Oracle8iDialect.java:450) [hibernate-core-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ...
    at org.hibernate.dialect.lock.PessimisticWriteSelectLockingStrategy.lock(PessimisticWriteSelectLockingStrategy.java:87) [hibernate-core-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ... 81 more
Caused by: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450) [ojdbc-7.jar:12.1.0.1.0]
    ...
    at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)
    at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) [hibernate-core-4.2.18.Final-redhat-2.jar:4.2.18.Final-redhat-2]
    ... 82 more

Oracle Trace Log:

    =====================
PARSING IN CURSOR #4 len=87 dep=0 uid=29 oct=3 lid=29 tim=1416963939942628 hv=3017806794 ad='48e71488'
select ORDER_ID from ORDER where ORDER_ID =:1  for update
END OF STMT
PARSE #4:c=0,e=7,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1416963939942625
EXEC #4:c=0,e=43,p=0,cr=3,cu=1,mis=0,r=0,dep=0,og=4,tim=1416963939942708
FETCH #4:c=0,e=9,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1416963939942747
STAT #4 id=1 cnt=1 pid=0 pos=1 obj=0 op='FOR UPDATE  (cr=3 pr=0 pw=0 time=35 us)'
STAT #4 id=2 cnt=1 pid=1 pos=1 obj=82422 op='INDEX UNIQUE SCAN PK_ORDER (cr=3 pr=0 pw=0 time=12 us)'
=====================
PARSING IN CURSOR #3 len=87 dep=0 uid=29 oct=3 lid=29 tim=1416963939943362 hv=3017806794 ad='48e71488'
select ORDER_ID from ORDER where ORDER_ID =:1  for update
END OF STMT
PARSE #3:c=0,e=8,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1416963939943359
*** 2015-12-24 15:31:45.629
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
                       ---------Blocker(s)--------  ---------Waiter(s)---------
Resource Name          process session holds waits  process session holds waits
TX-00160021-002790fd        47    1799     X             51    1856           X
TX-00190009-0018cd94        51    1856     X             47    1799           X
session 1799: DID 0001-002F-0000004F    session 1856: DID 0001-0033-00000FCE
session 1856: DID 0001-0033-00000FCE    session 1799: DID 0001-002F-0000004F
Rows waited on:
Session 1856: obj - rowid = 000141F4 - AAAUH0ABVAAAdiXAAT
  (dictionary objn - 82420, file - 85, block - 120983, slot - 19)
Session 1799: obj - rowid = 000141F4 - AAAUH0ABVAAAdjJAAT
  (dictionary objn - 82420, file - 85, block - 121033, slot - 19)
Information on the OTHER waiting sessions:
Session 1856:
  pid=51 serial=19150 audsid=69033382 user: 29/ORAUSER
  O/S info: user: xxxx, term: unknown, ospid: 1234, machine: XXXX
            program: JDBC Thin Client
  application name: JDBC Thin Client, hash value=0
  Current SQL Statement:

update ORDER set ... where ORDER_ID=:75 
End of information on OTHER waiting sessions.
Current SQL statement for this session:
select ORDER_ID from ORDER where ORDER_ID =:1  for update
===================================================
PROCESS STATE
-------------
Process global information:
     process: 0x15e4ac4d0, call: 0x141bfe118, xact: 0x15b8275d0, curses: 0x15d202070, usrses: 0x15d202070
  ----------------------------------------
  SO: 0x15e4ac4d0, type: 2, owner: (nil), flag: INIT/-/-/0x00
  (process) Oracle pid=47, calls cur/top: 0x141bfe118/0x141bfe118, flag: (0) -
            int error: 0, call error: 0, sess error: 0, txn error 0
  (post info) last post received: 0 0 193
              last post received-location: kgskbwt: posting new actses to run
              last process to post me: 15e4a9d48 167 0
              last post sent: 0 0 193
              last post sent-location: kgskbwt: posting new actses to run
              last process posted by me: 15e4a8d78 202 0
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0x15e812e98
    O/S info: user: oracle, term: UNKNOWN, ospid: 12914
    OSD pid info: Unix process pid: 12914, image: oracleprod@pddevdb01
Dump of memory from 0x000000015E3B6870 to 0x000000015E3B6A78

Environment: JBoss EAP 6, Oracle 10g, Hibernate 4.2, Quartz 1.6, and EJB 3.

Notes: Upgrading Oracle is not currently an option. Both DB and application transaction isolation levels have not been changed.

---

UPDATE: After debugging with two machines , we're able to retrieve the same results from the "select...for update" query. Given Transaction A runs the for update query and Transaction B runs it's for update query before Transaction A commits, it will wait for A to commit before returning results, but once A commits, B returns the same results as A -- even though the value selecting on (processId) has already been changed by A (to -1). If A commits before B runs it's "for update" query, the two queries get different results.

Answer 1

The problem can be visualized in this log:

update ORDER set ... where ORDER_ID=:75 
End of information on OTHER waiting sessions.
Current SQL statement for this session:
select ORDER_ID from ORDER where ORDER_ID =:1  for update

You are acquiring the lock but only flushing and committing the status at the end of the Quartz transaction, which might take time.

In the meanwhile, all those records are locked and no other transaction can select or update those records.

I think you should change the order of execution as follows:

• you retain the select for update logic to lock the entries you want to process and update the status to IN_PROGRESS
• you commit that transaction as soon as possible to release the locks
• you continue with the selected records that are now IN_PROGRESS in a new transaction
• the other transaction won't select the IN_PROGRESS records so that not two concurrent transaction process the same records