How often to call commit on an offline Solr/Lucene index?

Question

I know there have been some semi-similar questions, but in this case, I am building an index which is offline, until build is complete. I am building from scratch two cores, one has about 300k records with alot of citation information and large blocks of full text (this is the document index) and another core which has about 6.6 Million records, with full text (this is the page index).

Given this index is being built offline, the only real performance issue is speed of building. Noone should be querying this data.

The auto-commit would apparently fire if I stop adding items for 50 seconds? Which I don't do. I am adding ten at a time and they are added every couple seconds.

So, should I commit more often? I feel like the longer this runs the slower it gets, at least in my test case of 6k documents to index.

With noone searching this index, how often would anyone suggest I commit?

Should say I am using Solr 3.1 and SolrNet.

Answer 1

Although it's commits that are taking time for you, you might want to consider looking into other tweaking than commit frequency.

Is it the indexing core that also does searching, or is it replicated somewhere else after indexing concludes? If the latter is the case, then turning off caches might have a very noticeable impact on performance ( solr rebuilds caches every time you commit ).

Answer 2

You could also look into using the autoCommit or commitWith features of Solr.

• commitWithin is done as part of the document add command. I believe that this is supported with SolrNet - please see Using the commiWithin attribute thread for more details.
• autoCommit is a Solr configuration value added to the update handler section.