Lucene .Net, do i need to close IndexWriter

Question

We are having locking issues with Lucene .Net throwing a LockObtainFailedException . It is a multi tenanted site and each customer gets their own physical search index on disc, and a static list of IndexWriters is used, one per index to control changes.

We call the following functions on the IndexWriter

AddDocument();
DeleteDocuments();
DeleteAll();
Optimize();
Commit();

I have noticed that we never call Close() or Dispose() on the IndexWriter , and wanted to know if this was good practice and could be the cause of the issues.

Thanks Dave

Answer 1

The docs say yes, but only when you're killing off the application itself - otherwise, no. Here's the docs for IndexWriter.Dispose in Lucene.Net 4.8:

Commits all changes to an index, waits for pending merges to complete, and closes all associated files.

This is a "slow graceful shutdown" which may take a long time ...

Note that this may be a costly operation, so, try to re-use a single writer instead of closing and opening a new one. See Commit() for caveats about write caching done by some IO devices.

https://github.com/apache/lucenenet/blob/master/src/Lucene.Net/Index/IndexWriter.cs#L996

So, you should call .Dispose() , but, typically only once when you're shutting down the app. It is not however clear whether you need to Dispose() its underlying objects.

You're already calling .Commit() , which they recommend instead. I would guess your problem is actually related to threading. I'm just learning Lucene, but if I were in your position I'd try putting a standard .Net lock around any write calls to Lucene, so that only one thread has access to writes at a time. If it solves your issue, you know it was threading.

Locks are awfully painful, and Lucene writes may take a long time, so if the lock solves this issue it may introduce other problems like 2 threads attempting to write and one hanging or failing depending on how your code is written. If that does arise you'd probably want to implement a Write Queue so threads can quickly hand off what they'd like written to a cheap data structure like ConcurrentQueue, and then have those write ops startup the write operation if none is running, and keep dequeuing until everything's written out - then back to sleep.

Answer 2

To use Close/Dispose when you don't need the object any longer is always a good idea. There is a reason why a developer exposes these methods. Typically, the documentation give additional hints when to use these methods.

I also advise to use every IDisposeable -object in a using -block, which just calls Dispose() .

This gives objects the ability to clean up and free resources. In case of framework-objects this isn't really important since the garbage collector will care sooner or later, but in case of system-objects or handles like file-system handles Dispose becomes important. These handles might stay open.

In the case of the Lucene IndexWriter I'm not perfectly sure, but when it uses a file for its index (which is what I assume), then you have a reason why Dispose should be called.

When handles/connections/etc stay open it can lead to such exceptions. So, yes, you should use Close() / Dispose()