Lucene-内存不足错误

Question

I would like to store large amounts of file content (upwards of 75000 files with sizes around 5-100MB) in an index and run searches on it using Lucene 5. I'm using FSDirectory and I'm writing all file contents using an IndexWriter . As more files are written the memory usage increases until eventually an Out of Memory exception is thrown.

Here is an example of how I'm currently doing this.

Analyzer analyzer = new StandardAnalyzer();
FSDirectory directory = FSDirectory.open(indexFilePath);
DirectoryReader reader = DirectoryReader.open(directory);   

IndexWriterConfig config = new IndexWriterConfig(analyzer);

IndexWriter writer = new IndexWriter(directory, config);

for (Document document : documents)
{
    writer.addDocument(document);
}

writer.close();

I've been changing options around like these for the config but I've noticed no differences.

config.setMaxBufferedDocs(2);
config.setRAMBufferSizeMB(32);
config.setRAMPerThreadHardLimitMB(32);

I've also tried committing, flushing, and forcing merges with the writer but this doesn't affect it.

Is it possible to lower/limit the memory usage of Lucene?

Answer 1

You can Perform the lucene data indexing chunk by chunk. If you are full data indexing, perform the first chunk data indexing in IndexWriterConfig CREATE mode.

 config.setOpenMode(OpenMode.CREATE);

For indexing remaining chunks of data, set the IndexWriterConfig mode to CREATE_OR_APPEND

config.setOpenMode(OpenMode.CREATE_OR_APPEND);

This will perform incremental indexing, by appending the current data set to the existing lucene index.

Call these methods in each incremental indexing/chunk data indexing.

writer.optimize();
writer.commit();
writer.close(); 

TieredMergePolicy configuration can also be set explicitly only in the case of incremental indexing, for reflecting the delete, modidfication or addition of records to the index immediately on the search

TieredMergePolicy t  = new TieredMergePolicy();
t.setForceMergeDeletesPctAllowed(.01);
config.setMergePolicy(t);

writer.forceMergeDeletes();
writer.commit();

This is the way of doing indexing chunk by chunk. Since we are doing chunk by chunk. This will release the memory in each chunk.

Lucene indexing may or may not be the root cause of out of memory issue. Use Memory Analyzer tool to check which all java objects are not getting garbage collected in memory causing out of memory issue.