使用Apache Lucene索引大型文件时，如何解决内存不足错误？

Question

On line 195 of IndexFiles.java you will see:

 doc.add(new TextField("contents", new BufferedReader(new InputStreamReader(fis, StandardCharsets.UTF_8))));

This line allows for the user to search on file contents. If somebody wishes to display a summary along with the name of the matching file (kind of like Google search results) you need to add a some more lines of code after line 195 of IndexFiles.java as shown below:

FileReader fr = new FileReader("/home/user1/largefile.txt");
Bufferedreader  br = new BufferedReader(fr);

StringBuilder sb = new StringBuilder();
String line;

while ( (line = br.readLine()) != null){
   sb.append(line);
}

Field contentField = new StringField("content", sb.toString(), Field.Store.YES, Field.Index.ANALYZED); 

doc.add(contentField);

But I'm not done yet, I need to use Lucene's Highlighter class and add code after line 184 in SearchFiles.java . More specifically something like:

Document doc = searcher.doc(hits[i].doc);
String text = doc.getField("content");
highlighter = new Highlighter(new QueryScorer());
String summary = highlighter.getBestFragment(analyzer, "content", text);

This code works perfectly and gives me the summary of search results. However, if the files are too big the IndexFiles.java class spits out an OutOfMemeory error while appending to the StringBuilder(). How do I get around this?

Answer 1

The problem is that the java heap is exhausted, by default the maximum java heap size is 64MB, but you can increase it using the option Xmx ie -Xmx1g which increase the maximum heap size to 1GB, take into account that the amount of memory for the heap can't get over of the size of RAM.

If you want to know more take a look to this:

-X Command-line Options

How is the default java heap size determined?