Solr full-import performance

Question

I have a small set of queries and entities and even though the performance is pretty bad, I just would like to know what tricks and configurations that i can do to increase the performance ?

Note I'm using Solr 4.1.

Answer 1

You should try to minimize the number of commits during your import. Even if you don't commit periodically when adding documents to Solr, Solr will do an auto commit based on solrconfig.xml autoCommit settings:

<autoCommit>
   <maxDocs>10000</maxDocs>
   <maxTime>15000</maxTime>
   <openSearcher>false</openSearcher>
</autoCommit>

Increase both maxDocs and maxTime and see if you get better speeds. ( maxTime is in milli seconds, so default setting is 15 secs only, which is very low for bulk imports.)

You can even try disabling auto-commit during your bulk import and issue one commit command after all your documents are added. If this does not throw an out-of-memory exception from Solr, it is the best speed you can get.

If you were doing an RDBMS import, then I would have suggested capturing as many fields as possible using JOINs and minimizing the number of sub-entities, since each sub-entity opens a separate connection to the DB. Since you are importing from mongo, this doesn't apply to you. You can experiment by creating a new mongo collection with all the data you need for Solr, keep a single entity in your data importer and see if it improves import speed.