如何在Solr上优化大索引

Question

our index is rising relatively fast, by adding 2000-3000 documents a day. We are running an optimize every night.

The point is, that Solr needs double disc space while optimizing. Actually the index has an size of 44GB, which works on an 100GB partition - for the next few months.

The point is, that 50% of the disk space are unused for 90% of the day and only needed during optimize.

Next thing: we have to add more space on that partition periodical - which is always a painful discussion with the guys from the storage department (because we have more than one index...).

So the question is: is there a way to optimize an index without blocking additional 100% of the index size on disk?

I know, that multi-cores an distributed search is an option - but this is only an "fall back" solution, because for that we need to change the application basically.

Thank you!

Answer 1

There is continous merging going on under the hood in Lucene. Read up on the Merge Factor which can be set in the solrconfig.xml. If you tweak this setting you probably wont have to optimize at all.

Answer 2

You can try partial optimize by passing maxSegment parameter. This will reduce the index to that specified number. I suggest you do in batches (eg if there are 50 segments first reduce to 30 then to 15 and so on). Here's the url:

host:port/solr/CORE_NAME/update?optimize=true&maxSegments=(Enter the number of segments you want to reduce to. Ignore the parentheses)&waitFlush=false