Solr / Lucene中的大型文档索引中的可能问题

Question

I am trying to index a large data in solr/lucene. Since It is a legacy system and because of some other reasons, I have to do it via a C++ layer. But before doing that I wanted to optimize the process so I did google for that. I found out following things for that:

1. Indexing in batches: which will help me in scenario where indexing will fail in between because of some failure. So i can start with remaining batches again.
2. buffer lookup
3. indexer concurrency

I found the last 2 terms somewhere while looking for different issues, but I am unable to understand it fully.

So if anyone can help me in understanding these two issues and any other issue which may arise.

Answer 1

I'm not sure what you mean when you're mentioning " Buffer Lookup " - usually this is the case of allowing a server to have a decent in-memory cache, where as many queries as possible can be answered without having to recalculate the intersection between documents and which documents are contained in a certain set for each query. For Solr this is configured using the different *cache -settings. The requirements will be different for most applications, depending on query load, field definitions, etc. Performing a commit (making documents visible in the index) usually expires caches, as the cache might no longer be valid.

Indexer Concurrency allows a server to insert documents into the actual index from many threads at the same time, without locking between the threads. Lucene made concurrent indexing possible back in 2011 (for Lucene 4.0), and allows faster and more efficient updates of the index. Whether this matters depends on your application.