Out of memory while indexing with Lucene

Question

I'm using Lucene 4.9.0 to index 23k files, but now I'm receiving java.lang.OutOfMemoryError: Java heap space message . I don't want to increase "heap size" because the number of files tends to increase everyday. How can I index all files without the OOM problem and increase "heap space"?

Answer 1

Your question is too vague and makes little sense.

First of all, 23K files can be 1 byte/each or 1G/each. How are we supposed to know what's inside and how heavyweight they are?

Secondly, you say

I don't want to increase "heap size" because <...>

and straight after you say

How can I index all files without the OOM problem and increase "heap space"

Can you make up your mind on whether you can increase heap space or not?

There's a certain amount of memory required to index the data, and there's nothing much you can do about it. That said, the most memory required is during merging process and you can play with the merge factor to see if this helps you.