JBoss EAP 6.1 的 Infinispan 超时异常

Question

this one is driving me crazy. I am getting exceptions like this one at random points:

ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [60 seconds] on key [com.acme.entity.EntityA#9073] for requestor [GlobalTransaction:<null>:9593:local]! Lock held by [GlobalTransaction:<null>:9580:local]

The setup is fairly old: JBoss EAP 6.1 with Infinispan 5.2.6 for the 2nd level cache. There can be multiple JBoss server which run as standalone instances (not configured to be clustered) but they use SQLProxy and a percona MySQL cluster. (However we have seen the same problem on a single JBoss instance with one DB on the same server).

The configuration in standalone.xml is currently this:

  <subsystem xmlns="urn:jboss:domain:infinispan:1.5">
            <cache-container name="web" aliases="standard-session-cache" default-cache="local-web" module="org.jboss.as.clustering.web.infinispan">
                <local-cache name="local-web" batching="true">
                    <file-store passivation="false" purge="false"/>
                </local-cache>
            </cache-container>
            <cache-container name="hibernate" default-cache="local-query" module="org.jboss.as.jpa.hibernate:4">
                <local-cache name="entity">
                    <locking isolation="READ_COMMITTED" acquire-timeout="60000"/>
                    <transaction mode="NON_XA"/>
                    <eviction strategy="LRU" max-entries="10000"/>
                    <expiration max-idle="100000"/>
                </local-cache>
                <local-cache name="local-query">
                    <locking isolation="READ_COMMITTED" acquire-timeout="60000"/>
                    <transaction mode="NONE"/>
                    <eviction strategy="LRU" max-entries="10000"/>
                    <expiration max-idle="100000"/>
                </local-cache>
                <local-cache name="timestamps">
                    <transaction mode="NONE"/>
                    <eviction strategy="NONE"/>
                </local-cache>
            </cache-container>
        </subsystem>

We have tried different timeout values without any success. We have recently changed the isolation from the default REPEATABLE_READ (in JBoss 6.1) to READ_COMMITTED (which seems the default for Infinispan 5.2.6 and was changed to the default in WildFly 9) see also https://developer.jboss.org/thread/243458 . I was hoping that that would fix the problem but we are still seeing these Timeout exceptions.

For the entities that use 2nd level cache we use this on the entities:

@Cache(usage = CacheConcurrencyStrategy.TRANSACTIONAL,region="cache.StandardEntity")
public class EntityA

And in our infinispan.xml file we have:

 <namedCache name="Acme.cache.StandardEntity">
      <eviction strategy="LRU" maxEntries="1000"/>
      <expiration maxIdle="3600" lifespan="3600" wakeUpInterval="7200000"/>
   </namedCache>

The infispan.xml file was created by a tool when we migrated from EH cache a few years ago. The wakeUpInterval looks rather high. Could this be a problem? I am not actually sure whether the namedCache is configured correctly. Does it need to be prefixed with Acme (the name of the app in this case?). How can I test that these named Caches are actually used. I am a bit confused about what needs to be configured in the standalone.xml file and what in the infinispan.xml file.

Are there still things in the configuration that I could try to fix this problem?

If not how can I figure out who 'locked the door' (acquired the lock). I can see the threads trying to open the door (trying to acquire the lock) and complaining (the exception is thrown) but I cannot see who locked it in the first place. If I can see who is holding the lock for so long I might be able to fix it.

Locally I can enable logging for this

 <logger category="org.infinispan.util.concurrent.locks">
                <level name="TRACE"/>
 </logger>

but I can't really do this in production (too much logging). However locally I cannot reproduce it. Any other idea how I could find out which thread acquired the lock?

Any ideas are appreciated.

Thanks!

Answer 1

It seems you are having concurrent access problem on the lock. I would suggest pessimistic locking to avoid this problem, it increases the resources but it helps on those cases:

/subsystem=infinispan/cache-container=web/local-cache=persistent/transaction=TRANSACTION/:add(locking=PESSMISTIC)

On regards to infinispan.xml : this file is only for Red Hat Data GRid (JDG, RHDG) embedded mode (or the community Infinispan version). For EAP 6.1 you need to use the standalone-ha or standalone-full-ha profiles to set the cache properties.