How to use RAMDirectory and avoid OutOfMemoryException if the object size exceeds 2GB

Question

I have been putting my entire index into memory using RAMDirectory to improve performance and it worked beautifully until my index grew and grew. Now I am getting OutOfMemoryException . While my index on disk is 1.24GB, I suspect that the object size of the RAMDirectory object ends up exceeding the .NET 2GB object size limit and the exception is therefore thrown. Another reason might be that the virtual address space is simply too fragmented to find a hole big enough for my object.

I would love to continue using RAMDirectory . How can I do that while avoiding the OutOfMemoryException ?

Please also note that when I write my index I call IndexWriter.Optimize so the entire index is in one big file.

Answer 1

The only way to keep on using a RAMDirectory I can think of is to split it in several smaller indexes and use a MultiSearcher .

This way you'll be able to avoid the .NET 2GB object size limit, note that even on 64bit a single object still has a size limit of 2GB, RamDirectory holds an array of bytes internally to represent the index and thats probably whats making it blow up if its too big.

In my humble opinion tho, you should probably think about using a FSDirectory with large indexes, the speed is usually good enough for most applications after its warmed up.