使用带有特殊字符的字符串搜索记录在 Lucene.Net 中不起作用

Question

I am new to Lucene, here I am facing serious issues with lecene search . When searching records using string/string with numbers it's working fine. But it does not bring any results when search the records using a string with special characters.

ex: example - Brings results 
    'examples' - no result
    %example% - no result
    example2 - Brings results 
    @example - no  results

code:

Indexing;

_document.Add(new Field(dc.ColumnName, dr[dc.ColumnName].ToString(), Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.YES));

Search Query :

Lucene.Net.Store.Directory _dir = Lucene.Net.Store.FSDirectory.Open(Config.Get(directoryPath));
Lucene.Net.Analysis.Analyzer analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);

Query querySearch = queryParser.Parse("*" + searchParams.SearchForText + "*");
booleanQuery.Add(querySearch, Occur.MUST);

Can anyone help me to fix this.

Answer 1

It appears there's work to be done. I urge getting a good starter book on Lucene such as Lucene in Action, Second Edition (as you're using version 3). Although it targets Java examples are easily adapted to C#, and really the concepts are what matter most.

First, this:

"*" + searchParams.SearchForText + "*"

Don't do that . Leading wildcard searches are wildly inefficient and will suck an enormous amount of resources on a sizable index, doubly for leading and trailing wildcard search - what would happen if the query text was *e* ?

There also seems to be more going on than shown in posted code as there is no reason not to be getting hits based on the inputs. The snippet below will produce the following in the console:

Index terms:
example
example2

raw text %example% as query text:example got 1 hits
raw text 'example' as query text:example got 1 hits
raw text example as query text:example got 1 hits
raw text @example as query text:example got 1 hits
raw text example2 as query text:example2 got 1 hits
Wildcard raw text example* as query text:example* got 2 hit(s)

See the Index Terms listing? NO 'special characters' land in the index because StandardAnalyzer removes them at index time - assuming StandardAnalyzer is used to index the field?

I recommend running the snippet below in the debugger and observe what is happening.

public static void Example()
{
    var field_name = "text";
    var field_value = "%example% 'example' example @example example";
    var field_value2 = "example2";
    var luceneVer = Lucene.Net.Util.Version.LUCENE_30;

    using (var writer = new IndexWriter(new RAMDirectory(),
            new StandardAnalyzer(luceneVer), IndexWriter.MaxFieldLength.UNLIMITED)
            )
    {
        var doc = new Document();
        var field = new Field(
            field_name,
            field_value,
            Field.Store.YES,
            Field.Index.ANALYZED,
            Field.TermVector.YES
            );

        doc.Add(field);
        writer.AddDocument(doc);

        doc = new Document();
        field = new Field(
            field_name,
            field_value2,
            Field.Store.YES,
            Field.Index.ANALYZED,
            Field.TermVector.YES
            );

        doc.Add(field);
        writer.AddDocument(doc);
        writer.Commit();

        Console.WriteLine();
        // Show ALL terms in the index.
        using (var reader = writer.GetReader())
        {
            TermEnum terms = reader.Terms();
            Console.WriteLine("Index terms:");
            while (terms.Next())
            {
                Console.WriteLine("\t{0}", terms.Term.Text);
            }
        }

        // Search for each word in the original content @field_value
        using (var searcher = new IndexSearcher(writer.GetReader()))
        {
            string query_text;
            QueryParser parser;
            Query query;
            TopDocs topDocs;
            List<string> field_queries = new List<string>(field_value.Split(' '));
            field_queries.Add(field_value2);

            var analyzer = new StandardAnalyzer(luceneVer);
            while (field_queries.Count > 0)
            {
                query_text = field_queries[0];
                parser = new QueryParser(luceneVer, field_name, analyzer);
                query = parser.Parse(query_text);
                topDocs = searcher.Search(query, null, 100);
                Console.WriteLine();
                Console.WriteLine("raw text {0} as query {1} got {2} hit(s)",
                    query_text,
                    query,
                    topDocs.TotalHits
                    );
                field_queries.RemoveAt(0);
            }

            // Now do a wildcard query "example*"
            query_text = "example*";
            parser = new QueryParser(luceneVer, field_name, analyzer);
            query = parser.Parse(query_text);
            topDocs = searcher.Search(query, null, 100);
            Console.WriteLine();
            Console.WriteLine("Wildcard raw text {0} as query {1} got {2} hit(s)",
                query_text,
                query,
                topDocs.TotalHits
                );
        }
    }
}

If you need to perform exact matching, and index certain characters like %, then you'll need to use something other than StandardAnalyzer , perhaps a custom analyzer.