Deadlock - Threads Blocked in Java EE Web Application

Question

I hava a Java/Java EE Web Application, deployed in JBoss 6x.

Sometimes the application comes to halt, and there appears to be Deadlock situation, with many threads in BLOCKED state, as seen in this snippet from Thread Dump pasted at the end.

Questions :
1. What is a Transaction Reaper Worker - what resource has it locked?
2. Is this a JTA Issue? What is causing it. How can i investigate, resolve this further?
3. There are almost 10 Threads in BLOCKED state with similar stack trace. Could this problem have been caused by something going wrong at db end

Any help greatly appreciated.

"ajp-0.0.0.0-8809-19" - Thread t@238
java.lang.Thread.State: BLOCKED
    at com.arjuna.ats.arjuna.coordinator.BasicAction.removeChildThread(BasicAction.java:650)
    - waiting to lock <7c0d6> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction) owned by "Transaction Reaper Worker 44" t@942
    at com.arjuna.ats.internal.arjuna.thread.ThreadActionData.purgeActions(ThreadActionData.java:248
    ....
    ....
    Locked ownable synchronizers:
    - locked <ed3045> (a java.util.concurrent.locks.ReentrantLock$FairSync)

"Transaction Reaper Worker 44" - Thread t@942
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - waiting to lock <ed3045> (a java.util.concurrent.locks.ReentrantLock$FairSync) owned by "ajp-0.0.0.0-8809-19" t@238
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    .....
    at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2902)
    at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2881)
    at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1602)
    - locked <7c0d6> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
    at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:119)
    at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:212)
    at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:367)
    at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:79)     

Answer 1

This is a classic deadlock situation. Thread ajp-0.0.0.0-8809-19 has acquired a lock on object reference ed3045 and is waiting to acquire a lock on object reference 7c0d6 ; but Thread Transaction Reaper Worker 44 has the lock on the latter and is waiting to acquire the former. Neither is going to let go and let the other acquire their desired lock, so it's a deadlock.

To answer your questions:

1. A Transaction Reaper Worker is a Thread whose job is presumably to make sure that uncommitted transaction objects do not accumulate in the case of programmer error (eg not correctly closing transactions) or unhandled network errors
2. It's a bug in the JBoss JTA library (or possibly you're using it wrong, but it seems more like a bug). You should follow up with the JBoss team, preferably with some conditions to reproduce
3. It's likely to be due to an untested combination of error conditions, probably related to the network layer, or the DB as you surmise

Answer 2

It's certainly possible that this is a bug in JBoss Transactions. It's worth investigating whether it's a bug in your own code. The thread ajp-0.0.0.0-8809-19 is one of the servlet container's worker threads, which executes your code. The questionable action is its acquisition of the lock which the reaper later wants:

Locked ownable synchronizers:
- locked <ed3045> (a java.util.concurrent.locks.ReentrantLock$FairSync)

Can you identify the point in the stack where it locks that object? What is the closest code of yours, and what is it doing?