Lucene：NRT 性能非常慢

Question

I'm testing performance of NRT (near real time) searching in my application and i'm getting really odd results. I'm using this query as a sample to get all elements (the test set is very small so getting all elements shouldn't be an issue, this is 250 files getting indexed only, text files only, total index size= 1.5MB, the real set that i need to support is hundred of thouthands of files for a multi GB index)

Here's the sample query that is worrying me :

    public static List<IndexableItem> GetAllElements()
    {
        var qp = new Lucene.Net.QueryParsers.QueryParser(Lucene.Net.Util.Version.LUCENE_29, "ProviderPath", analyzer);
        qp.AllowLeadingWildcard = true;
        var query = qp.Parse("*");

        var searcher = new Lucene.Net.Search.IndexSearcher(reader);
        List<IndexableItem> docs = new List<IndexableItem>();
        searcher.Search(query, new SimpleHitCollector(docId =>
        {
            docs.Add(reader.Document(docId).ToIndexable());
        }));
        return docs;
    }

As you can see it's pretty simple. The run time of this query is around 0.1 second while indexing isn't running, but if i have indexing running at the same time it goes up to . . . . 45 seconds or more!

The reader variable is a property defined as such :

    public static IndexReader reader 
    {
        get 
        {
            return writer.GetReader();
        }
    }

And writer :

    static SearchIndexManager()
    {
        writer = new IndexWriter(FSDirectory.Open(@"C:\MyFolder"), analyzer, IndexWriter.MaxFieldLength.UNLIMITED);
    }

The performance issue is within lucene for sure (it's within the hitcollector, takes up to 1 second between each docs.Add line). The ToIndexable cannot be the issue either (it's a trivial method and is completely not dependant on anything the indexer could use (disk io etc).

I'm fairly sure something is wrong there as obviously the goal of NRT isn't to get a 450X slowdown, any advice on where i should look for hints?

Some more information : I'm not calling optimise during the slowdowns, and "once in a while" i'll get a quick answer even when indexing but it seems pretty random when that happens. I do am calling commit once in a while (every 100 insertions).

Answer 1

As I understand it, near-real-time search is intended for an index that has changed but changes have not yet been committed, and no further changes will occur while searching . This is the optimal usage of NRT, I don't mean that search is not possible at the same time as indexing. Searching or reading are suboptimal if they occur at the same time as indexing.

Consider the method IndexReader.Reopen . Its purpose is to obtain a fresh reader, if the index is changing or has changed since the old instance of IndexReader was obtained. Therefore if you continue using the old instance you might miss documents that you should find, and you are reading from a 'moving target', thus the slow performance.

You wrote:

index stays in 1 file and keeps growing (this is expected behavior), but as soon as i launch a search, it splits as if a commit was getting forced

When you get an IndexReader from the IndexWriter it will flush any buffered changes - note this is not a commit.