Why is the import process very slow in Solr 5.3.x?

Question

I'm using solr 5.3.1's DataImportHandler to import IMDB data which I imported into MySQL.

However it takes a couple of seconds even minutes to get one document processed. My table contains 10M+ rows so this is going to take forever. I have materialized all data and it only take a few minutes for MySQL to get all row processed.

What could have caused this poor performance?

Answer 1

@yangrui Unfortunately there is no single answer to your question on why indexing is slow. 24G is a lot of heap but depending on the actual size of your index it may or may not be enough.

Commit policy modification should also help in case you are committing too frequently. SOLR does a lot its magic of making documents available for searches when a 'commit' / 'autocommit' happens. However the when a commit does happen it is a resource hungry operation.

One other thing that is not obvious is the actual unallocated RAM available on the server. By unallocated I mean additional RAM on the server apart from the RAM that is associated with the JVM as Heap. I suggest going through this documentation https://wiki.apache.org/solr/SolrPerformanceProblems#RAM I suspect that you may not have enough RAM on your machine. Hope this helps.