TermQuery 没有给出预期的结果作为 QueryParser - Lucene 7.4.0

Question

I am indexing 10 text documents using StandardAnalyser.

public static void indexDoc(final IndexWriter writer, Path filePath, long timstamp)
    {
        try (InputStream iStream = Files.newInputStream(filePath))
        {
            Document doc = new Document();

            Field pathField = new StringField("path",filePath.toString(),Field.Store.YES);
            Field flagField = new TextField("ashish","i am stored",Field.Store.YES);
            LongPoint last_modi = new LongPoint("last_modified",timstamp);
            Field content = new TextField("content",new BufferedReader(new InputStreamReader(iStream,StandardCharsets.UTF_8)));

            doc.add(pathField);
            doc.add(last_modi);
            doc.add(content);
            doc.add(flagField);

            if(writer.getConfig().getOpenMode()==OpenMode.CREATE)
            {
                System.out.println("Adding "+filePath.toString());
                writer.addDocument(doc);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }



    } 

above is the code snippet used to index a document. for testing purpose, i am searching a field called as 'ashish'.

When I use QueryParser, Lucene gives the search results as expected.

public static void main(String[] args) throws Exception
    {
        String index = "E:\\Lucene\\Index";
        String field = "ashish";
        int hitsPerPage = 10;

        IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get(index)));
        IndexSearcher searcher = new IndexSearcher(reader);
        Analyzer analyzer = new StandardAnalyzer();

        QueryParser parser = new QueryParser(field, analyzer);

        String line = "i am stored";

        Query query = parser.parse(line);
      //  Query q = new TermQuery(new Term("ashish","i am stored"));
        System.out.println("Searching for: " + query.toString());



        TopDocs results = searcher.search(query, 5 * hitsPerPage);
        ScoreDoc[] hits = results.scoreDocs;

        int numTotalHits = Math.toIntExact(results.totalHits);
        System.out.println(numTotalHits + " total matching documents");

        for(int i=0;i<numTotalHits;i++)
        {
             Document doc = searcher.doc(hits[i].doc);
             String path = doc.get("path");
             String content = doc.get("ashish");
             System.out.println(path+"\n"+content);

        }



    } 

above code demonstrates the use of QueryParser to retrieve the desired field, which works properly. it hits all 10 documents, as i am storing this field for all 10 documents. all good here.

however when I use TermQuery API, I don't get the desired result. I am presenting the code change that I did for TermQuery.

public static void main(String[] args) throws Exception
    {
        String index = "E:\\Lucene\\Index";
        String field = "ashish";
        int hitsPerPage = 10;

        IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get(index)));
        IndexSearcher searcher = new IndexSearcher(reader);
        Analyzer analyzer = new StandardAnalyzer();

      //  QueryParser parser = new QueryParser(field, analyzer);

        String line = "i am stored";

       // Query query = parser.parse(line);
       Query q = new TermQuery(new Term("ashish","i am stored"));
        System.out.println("Searching for: " + q.toString());

        TopDocs results = searcher.search(q, 5 * hitsPerPage);
        ScoreDoc[] hits = results.scoreDocs;

        int numTotalHits = Math.toIntExact(results.totalHits);
        System.out.println(numTotalHits + " total matching documents");

        for(int i=0;i<numTotalHits;i++)
        {
             Document doc = searcher.doc(hits[i].doc);
             String path = doc.get("path");
             String content = doc.get("ashish");
             System.out.println(path+"\n"+content);
             System.out.println("----------------------------------------------------------------------------------");
        }



    }

also attaching the screenshot of TermQuery API execution.

did some research on stackoverflow itself example Lucene TermQuery and QueryParser but did not find any practical solution also the lucene version was very old in those examples.

would appreciate a help.

thanks in advance!

Answer 1

I got the answer of my question in this post link that explains how TermQuery works

TermQuery searches for entire String as it is. this behavior will give you improper results as while indexing data is often tokenized.

in the posted code, I was passing entire search String to TermQuery like
Query q = new TermQuery(new Term("ashish","i am stored"));
now in above case, Lucene is finding "i am stored" as it is, which it will never find because in indexing this string is tokenized.
instead I tried to search like Query q = new TermQuery(new Term("ashish","stored"));
Above query gave me an expected results.

thanks, Ashish

Answer 2

The real problem is your query string is not getting analyzed here. So, use same analyzer as used while indexing document and try using below code to analyze query string and then search.

IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get(index)));
IndexSearcher searcher = new IndexSearcher(reader);

QueryParser parser = new QueryParser("ashish", analyzer);
Query query = new TermQuery(new Term("ashish", "i am stored"));
query = parser.parse(query.toString());
ScoreDoc[] hits = searcher.search(query, 5).scoreDocs;