JMS ActiveMQ lock with Jenkins

Question

My project uses JMS with ActiveMQ and it works fine, both in production and when I run my tests local computer. My problem occurs when Jenkins runs my tests (the same ones that are successful in my computer), I get the following entry in the console output:

2016-11-17 12:17:19,277 INFO oaactivemq.store.SharedFileLocker Database activemq-data/localhost/KahaDB/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File 'activemq-data/localhost/KahaDB/lock' could not be locked.

Has anyone had the same experience and encountered a solution for the problem? Will be much grateful for your help!

More details of the console output:

2016-11-17 12:17:09,193 WARN oaabroker.jmx.ManagementContext Failed to start jmx connector: Cannot bind to URL [rmi://localhost:1099/jmxrmi]: javax.naming.NameAlreadyBoundException: jmxrmi [Root exception is java.rmi.AlreadyBoundException: jmxrmi]. Will restart management to re-create jmx connector, trying to remedy this issue.

2016-11-17 12:17:09,257 INFO oaastore.kahadb.plist.PListStore PListStore:[/var/lib/jenkins/jobs/mydir/workspace/myproject/activemq-data/localhost/tmp_storage] started

2016-11-17 12:17:09,271 INFO oaactivemq.broker.BrokerService Using Persistence Adapter: KahaDBPersistenceAdapter[/var/lib/jenkins/jobs/mydir/workspace/myproject/activemq-data/localhost/KahaDB]

2016-11-17 12:17:09,277 INFO oaactivemq.store.SharedFileLocker Database activemq-data/localhost/KahaDB/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File 'activemq-data/localhost/KahaDB/lock' could not be locked.

Answer 1

There is a previous ActiveMQ process still running that has locked the data store. This new instance you see is running as a "slave" and polling to acquire the lock.

Answer 2

I was able to figure out the problem. The build process running in Jenkins hanged during the testing phase and I was forced to abort it, which left a lock in the directory: dir-tests\\activemq-data\\localhost\\KahaDB. There are two ways to address this problem:

1) The most direct and correct solution is to delete dir-tests\\activemq-data\\localhost\\KahaDB\\lock, however I do not have access nor permissions in the remote Jenkins machine.

2) We can easily create the Jms broker with a different name, eg

 brokerService.setBrokerName("MyDummyBrokerName");

This causes Active-MQ to use the directory dir-tests\\activemq-data\\MyDummyBrokerName\\KahaDB, which means that it will use a different lock. However, this is a workaround and not a final solution because the old lock and corresponding directory will remain in Jenkins. Nevertheless, it solves the problem for the time being and the old locks can be deleted later.