IndexWriter.commit() not sufficient to change index

Question

I have just ported our Lucene implementation to 4.9 from 3.1, and I still can't seem to update documents in my index without a forceMerge(1) (used to be optimize). I understand that whenever I update a document the old document is marked deleted and the new document is stored in a new segment/file. The result I continue to see even after the upgrade is that the document is marked deleted, but the searchers never seem to see the new segment. Searches for that document are just empty. Why doesn't it work?

Code I expect to work:

IndexWriter indexWriter =
        new IndexWriter(FSDirectory.open(indexDirectoryFile),
            getIndexWriterConfig());
Document document = buildDocument(p);
indexWriter.updateDocument(
    new Term(SKU_FIELD_NAME, p.getSku().toString()), document);
indexWriter.commit();
indexWriter.close();
searcherManager.maybeRefresh();

Code that does work:

IndexWriter indexWriter =
        new IndexWriter(FSDirectory.open(indexDirectoryFile),
            getIndexWriterConfig());
Document document = buildDocument(p);
indexWriter.updateDocument(
    new Term(SKU_FIELD_NAME, p.getSku().toString()), document);
indexWriter.commit();
**indexWriter.forceMerge(1,true);**
indexWriter.close();
searcherManager.maybeRefresh();

Answer 1

The index has been changed after the commit, it's the SearcherManager which is giving you false impression.

What does the maybeRefresh() return? If false , it means that index is being refreshed by a different thread (which takes time for bigger indices). Until this refresh is finished, other threads simply have to live with the stale reader/searcher.

If you want to test this, open a new IndexReader explicitly (without SearcherManager ) and do a query after the commit() . I can bet you will see updated data in search results.