SOLR 4.6.0 索引并发问题通过 CLI - org.apache.solr.common.SolrException：索引锁定写入核心 XXX

Question

I have a web application running in Tomcat with a SQL database and SOLR backend.

The task is implementing a standalone enhanced version of a webapp triggered functionality. Existing Webapp version involves querying a specific product from a SQL database (which also exist in SOLR) and re-indexing/updating it in a SOLR engine.

I am running a main class in a JAR file deployed via CLI as follows:

java -Dxxx.base=/usr/local/xxx -cp "xxxutils-4.12-SNAPSHOT.jar:/usr/local/tomcat/lib/servlet-api.jar:/usr/local/tomcat/conf:/usr/local/xxx/conf:*" org.sam.xxx.utils.SolrReIndexExec

My CLI version is slightly different to the web, it queries all products from the SQL database and tries to re-Index all the products.

My program is failing when it tries to add the documents to SOLR with the following error:

//About to add document to EmbeddedSolrServer  

Exception in thread "main" org.apache.solr.common.SolrException: SolrCore 'XXXX' is not available due to init failure: Index locked for write for core XXXX
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:818)
        at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:110)
        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
        at org.test.sam.service.SolrService.addRecordToSolrCli(SolrService.java:270)
        at org.test.sam.service.SolrService.addProductToSolrCli(SolrService.java:517)
        at org.test.sam.function.ReIndexProductService.reIndexAllProducts(ReIndexProductService.java:106)
        at org.test.sam.utils.SolrReIndexExec.<init>(SolrReIndexExec.java:39)
        at org.test.sam.utils.SolrReIndexExec.main(SolrReIndexExec.java:52)
Caused by: org.apache.solr.common.SolrException: Index locked for write for core XXXX
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:625)
        at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:557)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:592)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked for write for core XXXX
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:491)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:755)
        ... 11 more

    -----------------------------------------------------------------------
    

After some investigation. I saw some post suggesting the following:

1. Changing the lockType to simple in the solrConfig.xml, that potential fix made no difference.
2. Deleting the write.lock file /usr/local/xxx/Data/solr/XXXX/data/index/write.lock and re-runing the JAR resulted in the error below:

Exception in thread "main" org.apache.solr.common.SolrException: SolrCore 'XXXX' is not available due to init failure: Error opening new searcher at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:818) at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:110) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) at org.apache.solr.client.solrj.SolrServer.add(So lrServer.java:102) at org.test.sam.service.SolrService.addRecordToSolrCli(SolrService.java:270) at org.test.sam.service.SolrService.addProductToSolrCli(SolrService.java:517) at org.test.sam.function.ReIndexProductService.reIndexAllProducts(ReIndexProductService.java:106) at org.test.sam.utils.SolrReIndexExec.(SolrReIndexExec.java:39) at org.test.sam.utils.SolrReIndexExec.main(SolrReIndexExec.java:52) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.(SolrCore.java:834) at org.apache.solr.core.SolrCore.(SolrCore.Z93F725A07423F E1C889F448B33D21F46Z:625) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:557) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:592) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.Z9 3F725A07423FE1C889F448B33D21F46Z:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589) at org.apache.solr.core.SolrCore.(SolrCore.Z93F725A07423FE1C889F448B33D21 F46Z:821)... 11 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/usr/local/pm6/Data/solr/XXXX/data/index/write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.(IndexWriter.java:702) at org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:77) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.Z93F725A07423FE1C889 F448B33D21F46Z:267) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)... 13 more

Our SOLR setup is a single SOLR Core per instance per machine. I think the root cause is that the SOLR instance is initialized via the webapp and I am trying to access it via CLI. Ideally, I would like to use the same instance created by the webapp.

Below is my code that create the solr server via CLI:

//Gets the solr server. 
public synchronized static EmbeddedSolrServer getSolrServerCli() {
    String basePath = "/usr/local/xxx/Data/solr";
    CoreContainer container = new CoreContainer(basePath);
    container.load();
    return new EmbeddedSolrServer(container, "XXXX");
}



//Then add document

 EmbeddedSolrServer server = SolrService.getSolrServerCli();

 UpdateResponse x = server.add(doc);

Any ideas will be appreciated. Thanks

Answer 1

My solution was using an existing web application entry point to interact with SOLR Index. I've implemented a standalone application triggered via CLI, which calls a servlet via a rest client. The servlet in turn uses a service class to interact with SOLR Index in a servlet context. Hence concurrency issue was avoided.

See my post here for more details.

How to Improve my Java App that migrates millions of records from MariaDB database to a SOLR Server

@MatsLindh - was my design buddy. Thanks Pal.